doc: Rewrite python onboarding tutorials in manual
Based on some feedback in #87094 and discussion with @fridh, this re-organizes the onboarding tutorial in the Nixpkgs manual's python section, so that we start with the simplest, most ad-hoc examples and work our way up. This progresses from: 1. How to create an temporary python env at the cmdline, then 2. How to create a specific python env for a single script, then 3. How to create a specific python env for a project in a shell.nix, then 4. How to install a specific python env globally on the system or in a user profile. Additionally, I've tried to standardize on some of the "best practice" ways of doing things: 1. Instead of saying that this command style is "supported but strongly not discouraged", I've just deleted it to avoid confusion. Bad: nix-shell -p python38Packages.numpy python38Packages.toolz Good: nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])' 2. In the portion where we show how to add stuff to the user's `XDG_CONFIG_HOME`, use overlays instead of `config.nix`. The former can do everything the latter can do, but is also much more generic and powerful, because it can compose with other files, compose with other envs, compose with overlays that do things like swap whether tensorflow and pytorch are built openblas/mkl/cuda stacks, and so on. The user is eventually going to see the overlay, so to avoid confusion let's standardize on it.
This commit is contained in:
parent
84765451c2
commit
c88290789c
@ -36,66 +36,288 @@ The Nix and NixOS manuals explain how packages are generally installed. In the
|
||||
case of Python and Nix, it is important to make a distinction between whether the
|
||||
package is considered an application or a library.
|
||||
|
||||
Applications on Nix are typically installed into your user
|
||||
profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the
|
||||
package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`.
|
||||
Dependencies such as libraries are automatically installed and should not be
|
||||
installed explicitly.
|
||||
Applications on Nix are typically installed into your user profile imperatively
|
||||
using `nix-env -i`, and on NixOS declaratively by adding the package name to
|
||||
`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
|
||||
such as libraries are automatically installed and should not be installed
|
||||
explicitly.
|
||||
|
||||
The same goes for Python applications and libraries. Python applications can be
|
||||
installed in your profile. But Python libraries you would like to use for
|
||||
development cannot be installed, at least not individually, because they won't
|
||||
be able to find each other resulting in import errors. Instead, it is possible
|
||||
to create an environment with `python.buildEnv` or `python.withPackages` where
|
||||
the interpreter and other executables are able to find each other and all of the
|
||||
modules.
|
||||
The same goes for Python applications. Python applications can be installed in
|
||||
your profile, and will be wrapped to find their exact library dependencies,
|
||||
without impacting other applications or polluting your user environment.
|
||||
|
||||
In the following examples we create an environment with Python 3.8, `numpy` and
|
||||
`toolz`. As you may imagine, there is one limitation here, and that's that
|
||||
you can install only one environment at a time. You will notice the complaints
|
||||
about collisions when you try to install a second environment.
|
||||
But Python libraries you would like to use for development cannot be installed,
|
||||
at least not individually, because they won't be able to find each other
|
||||
resulting in import errors. Instead, it is possible to create an environment
|
||||
with `python.buildEnv` or `python.withPackages` where the interpreter and other
|
||||
executables are wrapped to be able to find each other and all of the modules.
|
||||
|
||||
##### Environment defined in separate `.nix` file
|
||||
In the following examples we will start by creating a simple, ad-hoc environment
|
||||
with a nix-shell that has `numpy` and `toolz` in Python 3.8; then we will create
|
||||
a re-usable environment in a single-file Python script; then we will create a
|
||||
full Python environment for development with this same environment.
|
||||
|
||||
Philosphically, this should be familiar to users who are used to a `venv` style
|
||||
of development: individual projects create their own Python environments without
|
||||
impacting the global environment or each other.
|
||||
|
||||
#### Ad-hoc temporary Python environment with `nix-shell`
|
||||
|
||||
The simplest way to start playing with the way nix wraps and sets up Python
|
||||
environments is with `nix-shell` at the cmdline. These environments create a
|
||||
temporary shell session with a Python and a *precise* list of packages (plus
|
||||
their runtime dependencies), with no other Python packages in the Python
|
||||
interpreter's scope.
|
||||
|
||||
To create a Python 3.8 session with `numpy` and `toolz` available, run:
|
||||
|
||||
```sh
|
||||
$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])'
|
||||
```
|
||||
|
||||
By default `nix-shell` will start a `bash` session with this interpreter in our
|
||||
`PATH`, so if we then run:
|
||||
|
||||
```
|
||||
[nix-shell:~/src/nixpkgs]$ python3
|
||||
Python 3.8.1 (default, Dec 18 2019, 19:06:26)
|
||||
[GCC 9.2.0] on linux
|
||||
Type "help", "copyright", "credits" or "license" for more information.
|
||||
>>> import numpy; import toolz
|
||||
```
|
||||
|
||||
Note that no other modules are in scope, even if they were imperatively
|
||||
installed into our user environment as a dependency of a Python application:
|
||||
|
||||
```
|
||||
>>> import requests
|
||||
Traceback (most recent call last):
|
||||
File "<stdin>", line 1, in <module>
|
||||
ModuleNotFoundError: No module named 'requests'
|
||||
```
|
||||
|
||||
We can add as many additional modules onto the `nix-shell` as we need, and we
|
||||
will still get 1 wrapped Python interpreter. We can start the interpreter
|
||||
directly like so:
|
||||
|
||||
```sh
|
||||
$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz requests ])' --run python3
|
||||
these derivations will be built:
|
||||
/nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv
|
||||
building '/nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv'...
|
||||
created 277 symlinks in user environment
|
||||
Python 3.8.1 (default, Dec 18 2019, 19:06:26)
|
||||
[GCC 9.2.0] on linux
|
||||
Type "help", "copyright", "credits" or "license" for more information.
|
||||
>>> import requests
|
||||
>>>
|
||||
```
|
||||
|
||||
Notice that this time it built a new Python environment, which now includes
|
||||
`requests`. Building an environment just creates wrapper scripts that expose the
|
||||
selected dependencies to the interpreter while re-using the actual modules. This
|
||||
means if any other env has installed `requests` or `numpy` in a different
|
||||
context, we don't need to recompile them -- we just recompile the wrapper script
|
||||
that sets up an interpreter pointing to them. This matters much more for "big"
|
||||
modules like `pytorch` or `tensorflow`.
|
||||
|
||||
Module names usually match their names on [pypi.org](https://pypi.org/), but
|
||||
you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
|
||||
to find them as well (along with non-python packages).
|
||||
|
||||
At this point we can create throwaway experimental Python environments with
|
||||
arbitrary dependencies. This is a good way to get a feel for how the Python
|
||||
interpreter and dependencies work in Nix and NixOS, but to do some actual
|
||||
development, we'll want to make it a bit more persistent.
|
||||
|
||||
##### Running Python scripts and using `nix-shell` as shebang
|
||||
|
||||
Sometimes, we have a script whose header looks like this:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
import numpy as np
|
||||
a = np.array([1,2])
|
||||
b = np.array([3,4])
|
||||
print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
|
||||
```
|
||||
|
||||
Executing this script requires a `python3` that has `numpy`. Using what we learned
|
||||
in the previous section, we could startup a shell and just run it like so:
|
||||
|
||||
```
|
||||
nix-shell -p 'python38.withPackages(ps: with ps; [ numpy ])' --run 'python3 foo.py'
|
||||
The dot product of [1 2] and [3 4] is: 11
|
||||
```
|
||||
|
||||
But if we maintain the script ourselves, and if there are more dependencies, it
|
||||
may be nice to encode those dependencies in source to make the script re-usable
|
||||
without that bit of knowledge. That can be done by using `nix-shell` as a
|
||||
[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix), like so:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env nix-shell
|
||||
#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
|
||||
import numpy as np
|
||||
a = np.array([1,2])
|
||||
b = np.array([3,4])
|
||||
print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
|
||||
```
|
||||
|
||||
Then we simply execute it, without requiring any environment setup at all!
|
||||
|
||||
```sh
|
||||
$ ./foo.py
|
||||
The dot product of [1 2] and [3 4] is: 11
|
||||
```
|
||||
|
||||
If the dependencies are not available on the host where `foo.py` is executed, it
|
||||
will build or download them from a Nix binary cache prior to starting up, prior
|
||||
that it is executed on a machine with a multi-user nix installation.
|
||||
|
||||
This provides a way to ship a self bootstrapping Python script, akin to a
|
||||
statically linked binary, where it can be run on any machine (provided nix is
|
||||
installed) without having to assume that `numpy` is installed globally on the
|
||||
system.
|
||||
|
||||
By default it is pulling the import checkout of Nixpkgs itself from our nix
|
||||
channel, which is nice as it cache aligns with our other package builds, but we
|
||||
can make it fully reproducible by pinning the `nixpkgs` import:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env nix-shell
|
||||
#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
|
||||
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d373d80b1207d52621961b16aa4a3438e4f98167.tar.gz
|
||||
import numpy as np
|
||||
a = np.array([1,2])
|
||||
b = np.array([3,4])
|
||||
print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
|
||||
```
|
||||
|
||||
This will execute with the exact same versions of Python 3.8, numpy, and system
|
||||
dependencies a year from now as it does today, because it will always use
|
||||
exactly git commit `d373d80b1207d52621961b16aa4a3438e4f98167` of Nixpkgs for all
|
||||
of the package versions.
|
||||
|
||||
This is also a great way to ensure the script executes identically on different
|
||||
servers.
|
||||
|
||||
##### Load environment from `.nix` expression
|
||||
|
||||
We've now seen how to create an ad-hoc temporary shell session, and how to
|
||||
create a single script with Python dependencies, but in the course of normal
|
||||
development we're usually working in an entire package repository.
|
||||
|
||||
As explained in the Nix manual, `nix-shell` can also load an expression from a
|
||||
`.nix` file. Say we want to have Python 3.8, `numpy` and `toolz`, like before,
|
||||
in an environment. We can add a `shell.nix` file describing our dependencies:
|
||||
|
||||
Create a file, e.g. `build.nix`, with the following expression
|
||||
```nix
|
||||
with import <nixpkgs> {};
|
||||
|
||||
python38.withPackages (ps: with ps; [ numpy toolz ])
|
||||
(python38.withPackages (ps: [ps.numpy ps.toolz])).env
|
||||
```
|
||||
and install it in your profile with
|
||||
```shell
|
||||
nix-env -if build.nix
|
||||
```
|
||||
Now you can use the Python interpreter, as well as the extra packages (`numpy`,
|
||||
`toolz`) that you added to the environment.
|
||||
|
||||
##### Environment defined in `~/.config/nixpkgs/config.nix`
|
||||
And then at the command line, just typing `nix-shell` produces the same
|
||||
environment as before. In a normal project, we'll likely have many more
|
||||
dependencies; this can provide a way for developers to share the environments
|
||||
with each other and with CI builders.
|
||||
|
||||
If you prefer you could also add the environment as a package override to the
|
||||
Nixpkgs set, e.g. using `config.nix`,
|
||||
What's happening here?
|
||||
|
||||
1. We begin with importing the Nix Packages collections. `import <nixpkgs>`
|
||||
imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
|
||||
brings all attributes of `nixpkgs` in the local scope. These attributes form
|
||||
the main package set.
|
||||
2. Then we create a Python 3.8 environment with the `withPackages` function, as before.
|
||||
3. The `withPackages` function expects us to provide a function as an argument
|
||||
that takes the set of all Python packages and returns a list of packages to
|
||||
include in the environment. Here, we select the packages `numpy` and `toolz`
|
||||
from the package set.
|
||||
|
||||
To combine this with `mkShell` you can:
|
||||
|
||||
```nix
|
||||
{ # ...
|
||||
with import <nixpkgs> {};
|
||||
let
|
||||
pythonEnv = python38.withPackages (ps: [
|
||||
ps.numpy
|
||||
ps.toolz
|
||||
]);
|
||||
in mkShell {
|
||||
buildInputs = [
|
||||
pythonEnv
|
||||
|
||||
packageOverrides = pkgs: with pkgs; {
|
||||
myEnv = python38.withPackages (ps: with ps; [ numpy toolz ]);
|
||||
black
|
||||
mypy
|
||||
|
||||
libffi
|
||||
openssl
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
This will create a unified environment that has not just our Python interpreter
|
||||
and its Python dependencies, but also tools like `black` or `mypy` and libraries
|
||||
like `libffi` the `openssl` in scope. This is generic and can span any number of
|
||||
tools or languages across the Nixpkgs ecosystem.
|
||||
|
||||
##### Installing environments globally on the system
|
||||
|
||||
Up to now, we've been creating environments scoped to an ad-hoc shell session,
|
||||
or a single script, or a single project. This is generally advisable, as it
|
||||
avoids pollution across contexts.
|
||||
|
||||
However, sometimes we know we will often want a Python with some basic packages,
|
||||
and want this available without having to enter into a shell or build context.
|
||||
This can be useful to have things like vim/emacs editors and plugins or shell
|
||||
tools "just work" without having to set them up, or when running other software
|
||||
that expects packages to be installed globally.
|
||||
|
||||
To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
|
||||
that looks like this:
|
||||
|
||||
```nix
|
||||
# ~/.config/nixpkgs/overlays/myEnv.nix
|
||||
self: super: {
|
||||
myEnv = super.buildEnv {
|
||||
name = "myEnv";
|
||||
paths = [
|
||||
# A Python 3 interpreter with some packages
|
||||
(self.python3.withPackages (
|
||||
ps: with ps; [
|
||||
pyflakes
|
||||
pytest
|
||||
python-language-server
|
||||
]
|
||||
))
|
||||
|
||||
# Some other packages we'd like as part of this env
|
||||
self.mypy
|
||||
self.black
|
||||
self.ripgrep
|
||||
self.tmux
|
||||
];
|
||||
};
|
||||
}
|
||||
```
|
||||
and install it in your profile with
|
||||
|
||||
```shell
|
||||
nix-env -iA nixpkgs.myEnv
|
||||
You can then build and install this to your profile with:
|
||||
|
||||
```sh
|
||||
nix-env -iA myEnv
|
||||
```
|
||||
|
||||
The environment is is installed by referring to the attribute, and considering
|
||||
the `nixpkgs` channel was used.
|
||||
One limitation of this is that you can only have 1 Python env installed
|
||||
globally, since they conflict on the `python` to load out of your `PATH`.
|
||||
|
||||
If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
|
||||
atomically *uninstall* all other imperatively installed packages and replace
|
||||
your profile with just `myEnv` by using the `--replace` flag.
|
||||
|
||||
##### Environment defined in `/etc/nixos/configuration.nix`
|
||||
|
||||
For the sake of completeness, here's another example how to install the
|
||||
environment system-wide.
|
||||
For the sake of completeness, here's how to install the environment system-wide
|
||||
on NixOS.
|
||||
|
||||
```nix
|
||||
{ # ...
|
||||
@ -106,121 +328,16 @@ environment system-wide.
|
||||
}
|
||||
```
|
||||
|
||||
#### Temporary Python environment with `nix-shell`
|
||||
|
||||
The examples in the previous section showed how to install a Python environment
|
||||
into a profile. For development you may need to use multiple environments.
|
||||
`nix-shell` gives the possibility to temporarily load another environment, akin
|
||||
to `virtualenv`.
|
||||
|
||||
There are two methods for loading a shell with Python packages. The first and
|
||||
recommended method is to create an environment with `python.buildEnv` or
|
||||
`python.withPackages` and load that. E.g.
|
||||
|
||||
```sh
|
||||
$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])'
|
||||
```
|
||||
|
||||
opens a shell from which you can launch the interpreter
|
||||
|
||||
```sh
|
||||
[nix-shell:~] python3
|
||||
```
|
||||
|
||||
The other method, which is not recommended, does not create an environment and
|
||||
requires you to list the packages directly,
|
||||
|
||||
```sh
|
||||
$ nix-shell -p python38.pkgs.numpy python38.pkgs.toolz
|
||||
```
|
||||
|
||||
Again, it is possible to launch the interpreter from the shell. The Python
|
||||
interpreter has the attribute `pkgs` which contains all Python libraries for
|
||||
that specific interpreter.
|
||||
|
||||
##### Load environment from `.nix` expression
|
||||
As explained in the Nix manual, `nix-shell` can also load an
|
||||
expression from a `.nix` file. Say we want to have Python 3.8, `numpy`
|
||||
and `toolz`, like before, in an environment. Consider a `shell.nix` file
|
||||
with
|
||||
|
||||
```nix
|
||||
with import <nixpkgs> {};
|
||||
|
||||
(python38.withPackages (ps: [ps.numpy ps.toolz])).env
|
||||
```
|
||||
|
||||
Executing `nix-shell` gives you again a Nix shell from which you can run Python.
|
||||
|
||||
What's happening here?
|
||||
|
||||
1. We begin with importing the Nix Packages collections. `import <nixpkgs>`
|
||||
imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
|
||||
brings all attributes of `nixpkgs` in the local scope. These attributes form
|
||||
the main package set.
|
||||
2. Then we create a Python 3.8 environment with the `withPackages` function.
|
||||
3. The `withPackages` function expects us to provide a function as an argument
|
||||
that takes the set of all python packages and returns a list of packages to
|
||||
include in the environment. Here, we select the packages `numpy` and `toolz`
|
||||
from the package set.
|
||||
|
||||
To combine this with `mkShell` you can:
|
||||
|
||||
```nix
|
||||
with import <nixpkgs> {};
|
||||
|
||||
let
|
||||
pythonEnv = python38.withPackages (ps: [
|
||||
ps.numpy
|
||||
ps.toolz
|
||||
]);
|
||||
in mkShell {
|
||||
buildInputs = [
|
||||
pythonEnv
|
||||
hello
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
##### Execute command with `--run`
|
||||
A convenient option with `nix-shell` is the `--run`
|
||||
option, with which you can execute a command in the `nix-shell`. We can
|
||||
e.g. directly open a Python shell
|
||||
|
||||
```sh
|
||||
$ nix-shell -p python38Packages.numpy python38Packages.toolz --run "python3"
|
||||
```
|
||||
|
||||
or run a script
|
||||
|
||||
```sh
|
||||
$ nix-shell -p python38Packages.numpy python38Packages.toolz --run "python3 myscript.py"
|
||||
```
|
||||
|
||||
##### `nix-shell` as shebang
|
||||
In fact, for the second use case, there is a more convenient method. You can add
|
||||
a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script
|
||||
specifying which dependencies `nix-shell` needs. With the following shebang, you
|
||||
can just execute `./myscript.py`, and it will make available all dependencies
|
||||
and run the script in the `python3` shell.
|
||||
|
||||
```py
|
||||
#! /usr/bin/env nix-shell
|
||||
#! nix-shell -i python3 -p "python3.withPackages(ps: [ps.numpy])"
|
||||
|
||||
import numpy
|
||||
|
||||
print(numpy.__version__)
|
||||
```
|
||||
|
||||
### Developing with Python
|
||||
|
||||
Now that you know how to get a working Python environment with Nix, it is time
|
||||
to go forward and start actually developing with Python. We will first have a
|
||||
look at how Python packages are packaged on Nix. Then, we will look at how you
|
||||
can use development mode with your code.
|
||||
Above, we were mostly just focused on use cases and what to do to get started
|
||||
creating working Python environments in nix.
|
||||
|
||||
#### Packaging a library
|
||||
Now that you know the basics to be up and running, it is time to take a step
|
||||
back and take a deeper look at at how Python packages are packaged on Nix. Then,
|
||||
we will look at how you can use development mode with your code.
|
||||
|
||||
#### Python library packages in Nixpkgs
|
||||
|
||||
With Nix all packages are built by functions. The main function in Nix for
|
||||
building Python libraries is `buildPythonPackage`. Let's see how we can build the
|
||||
@ -262,6 +379,7 @@ An expression for `toolz` can be found in the Nixpkgs repository. As explained
|
||||
in the introduction of this Python section, a derivation of `toolz` is available
|
||||
for each interpreter version, e.g. `python38.pkgs.toolz` refers to the `toolz`
|
||||
derivation corresponding to the CPython 3.8 interpreter.
|
||||
|
||||
The above example works when you're directly working on
|
||||
`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
|
||||
you will want to test a Nix expression outside of the Nixpkgs tree.
|
||||
@ -312,7 +430,7 @@ Our example, `toolz`, does not have any dependencies on other Python packages or
|
||||
system libraries. According to the manual, `buildPythonPackage` uses the
|
||||
arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If
|
||||
something is exclusively a build-time dependency, then the dependency should be
|
||||
included as a `buildInput`, but if it is (also) a runtime dependency, then it
|
||||
included in `buildInputs`, but if it is (also) a runtime dependency, then it
|
||||
should be added to `propagatedBuildInputs`. Test dependencies are considered
|
||||
build-time dependencies and passed to `checkInputs`.
|
||||
|
||||
@ -423,10 +541,11 @@ Note also the line `doCheck = false;`, we explicitly disabled running the test-s
|
||||
|
||||
#### Develop local package
|
||||
|
||||
As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode) (`python setup.py develop`);
|
||||
instead of installing the package this command creates a special link to the project code.
|
||||
That way, you can run updated code without having to reinstall after each and every change you make.
|
||||
Development mode is also available. Let's see how you can use it.
|
||||
As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
|
||||
(`python setup.py develop`); instead of installing the package this command
|
||||
creates a special link to the project code. That way, you can run updated code
|
||||
without having to reinstall after each and every change you make. Development
|
||||
mode is also available. Let's see how you can use it.
|
||||
|
||||
In the previous Nix expression the source was fetched from an url. We can also
|
||||
refer to a local source instead using `src = ./path/to/source/tree;`
|
||||
@ -455,7 +574,6 @@ buildPythonPackage rec {
|
||||
It is important to note that due to how development mode is implemented on Nix
|
||||
it is not possible to have multiple packages simultaneously in development mode.
|
||||
|
||||
|
||||
### Organising your packages
|
||||
|
||||
So far we discussed how you can use Python on Nix, and how you can develop with
|
||||
@ -481,11 +599,11 @@ We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
|
||||
|
||||
buildPythonPackage rec {
|
||||
pname = "toolz";
|
||||
version = "0.7.4";
|
||||
version = "0.10.0";
|
||||
|
||||
src = fetchPypi {
|
||||
inherit pname version;
|
||||
sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
|
||||
sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
|
||||
};
|
||||
|
||||
meta = with lib; {
|
||||
@ -497,8 +615,8 @@ buildPythonPackage rec {
|
||||
}
|
||||
```
|
||||
|
||||
It takes an argument `buildPythonPackage`.
|
||||
We now call this function using `callPackage` in the definition of our environment
|
||||
It takes an argument `buildPythonPackage`. We now call this function using
|
||||
`callPackage` in the definition of our environment
|
||||
|
||||
```nix
|
||||
with import <nixpkgs> {};
|
||||
@ -548,7 +666,7 @@ Each interpreter has the following attributes:
|
||||
- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
|
||||
- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
|
||||
- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
|
||||
- `executable`. Name of the interpreter executable, e.g. `python3.7`.
|
||||
- `executable`. Name of the interpreter executable, e.g. `python3.8`.
|
||||
- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
|
||||
|
||||
### Building packages and applications
|
||||
@ -643,7 +761,7 @@ following are specific to `buildPythonPackage`:
|
||||
appears more than once in dependency tree. Default is `true`.
|
||||
* `disabled` ? false: If `true`, package is not built for the particular Python
|
||||
interpreter version.
|
||||
* `dontWrapPythonPrograms ? false`: Skip wrapping of python programs.
|
||||
* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
|
||||
* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
|
||||
variable in wrapped programs.
|
||||
* `installFlags ? []`: A list of strings. Arguments to be passed to `pip
|
||||
@ -730,7 +848,7 @@ Another difference is that `buildPythonPackage` by default prefixes the names of
|
||||
the packages with the version of the interpreter. Because this is irrelevant for
|
||||
applications, the prefix is omitted.
|
||||
|
||||
When packaging a python application with `buildPythonApplication`, it should be
|
||||
When packaging a Python application with `buildPythonApplication`, it should be
|
||||
called with `callPackage` and passed `python` or `pythonPackages` (possibly
|
||||
specifying an interpreter version), like this:
|
||||
|
||||
@ -761,7 +879,7 @@ luigi = callPackage ../applications/networking/cluster/luigi { };
|
||||
```
|
||||
|
||||
Since the package is an application, a consumer doesn't need to care about
|
||||
python versions or modules, which is why they don't go in `pythonPackages`.
|
||||
Python versions or modules, which is why they don't go in `pythonPackages`.
|
||||
|
||||
#### `toPythonApplication` function
|
||||
|
||||
@ -875,7 +993,7 @@ thus be also written like this:
|
||||
```nix
|
||||
with import <nixpkgs> {};
|
||||
|
||||
(python36.withPackages (ps: [ps.numpy ps.requests])).env
|
||||
(python38.withPackages (ps: [ps.numpy ps.requests])).env
|
||||
```
|
||||
|
||||
In contrast to `python.buildEnv`, `python.withPackages` does not support the
|
||||
@ -932,7 +1050,7 @@ pythonPackages.buildPythonPackage {
|
||||
Running `nix-shell` with no arguments should give you the environment in which
|
||||
the package would be built with `nix-build`.
|
||||
|
||||
Shortcut to setup environments with C headers/libraries and python packages:
|
||||
Shortcut to setup environments with C headers/libraries and Python packages:
|
||||
|
||||
```shell
|
||||
nix-shell -p pythonPackages.pyramid zlib libjpeg git
|
||||
@ -963,7 +1081,6 @@ have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
|
||||
and [PYTHONHASHSEED=0](https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONHASHSEED).
|
||||
Both are also exported in `nix-shell`.
|
||||
|
||||
|
||||
### Automatic tests
|
||||
|
||||
It is recommended to test packages as part of the build process.
|
||||
@ -976,7 +1093,7 @@ example of such a situation is when `py.test` is used.
|
||||
#### Common issues
|
||||
|
||||
* Non-working tests can often be deselected. By default `buildPythonPackage`
|
||||
runs `python setup.py test`. Most python modules follows the standard test
|
||||
runs `python setup.py test`. Most Python modules follows the standard test
|
||||
protocol where the pytest runner can be used instead. `py.test` supports a
|
||||
`-k` parameter to ignore test methods or classes:
|
||||
|
||||
@ -1052,7 +1169,7 @@ let
|
||||
newpkgs = import pkgs.path { overlays = [ (self: super: {
|
||||
python38 = let
|
||||
packageOverrides = python-self: python-super: {
|
||||
numpy = python-super.numpy_1_18.3;
|
||||
numpy = python-super.numpy_1_18;
|
||||
};
|
||||
in super.python38.override {inherit packageOverrides;};
|
||||
} ) ]; };
|
||||
@ -1127,14 +1244,14 @@ If you want to create a Python environment for development, then the recommended
|
||||
method is to use `nix-shell`, either with or without the `python.buildEnv`
|
||||
function.
|
||||
|
||||
### How to consume python modules using pip in a virtual environment like I am used to on other Operating Systems?
|
||||
### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems?
|
||||
|
||||
While this approach is not very idiomatic from Nix perspective, it can still be
|
||||
useful when dealing with pre-existing projects or in situations where it's not
|
||||
feasible or desired to write derivations for all required dependencies.
|
||||
|
||||
This is an example of a `default.nix` for a `nix-shell`, which allows to consume
|
||||
a virtual environment created by `venv`, and install python modules through
|
||||
a virtual environment created by `venv`, and install Python modules through
|
||||
`pip` the traditional way.
|
||||
|
||||
Create this `default.nix` file, together with a `requirements.txt` and simply
|
||||
@ -1149,7 +1266,7 @@ in pkgs.mkShell rec {
|
||||
name = "impurePythonEnv";
|
||||
venvDir = "./.venv";
|
||||
buildInputs = [
|
||||
# A python interpreter including the 'venv' module is required to bootstrap
|
||||
# A Python interpreter including the 'venv' module is required to bootstrap
|
||||
# the environment.
|
||||
pythonPackages.python
|
||||
|
||||
@ -1163,7 +1280,7 @@ in pkgs.mkShell rec {
|
||||
pythonPackages.requests
|
||||
|
||||
# In this particular example, in order to compile any binary extensions they may
|
||||
# require, the python modules listed in the hypothetical requirements.txt need
|
||||
# require, the Python modules listed in the hypothetical requirements.txt need
|
||||
# the following packages to be installed locally:
|
||||
taglib
|
||||
openssl
|
||||
@ -1183,7 +1300,7 @@ in pkgs.mkShell rec {
|
||||
}
|
||||
```
|
||||
|
||||
In case the supplied venvShellHook is insufficient, or when python 2 support is
|
||||
In case the supplied venvShellHook is insufficient, or when Python 2 support is
|
||||
needed, you can define your own shell hook and adapt to your needs like in the
|
||||
following example:
|
||||
|
||||
@ -1229,7 +1346,7 @@ in pkgs.mkShell rec {
|
||||
```
|
||||
|
||||
Note that the `pip install` is an imperative action. So every time `nix-shell`
|
||||
is executed it will attempt to download the python modules listed in
|
||||
is executed it will attempt to download the Python modules listed in
|
||||
requirements.txt. However these will be cached locally within the `virtualenv`
|
||||
folder and not downloaded again.
|
||||
|
||||
@ -1290,9 +1407,8 @@ self: super: {
|
||||
|
||||
### How to use Intel's MKL with numpy and scipy?
|
||||
|
||||
MKL can be configured using an overlay. See the section “[Using
|
||||
overlays to configure
|
||||
alternatives](#sec-overlays-alternatives-blas-lapack)”.
|
||||
MKL can be configured using an overlay. See the section "[Using overlays to
|
||||
configure alternatives](#sec-overlays-alternatives-blas-lapack)".
|
||||
|
||||
### What inputs do `setup_requires`, `install_requires` and `tests_require` map to?
|
||||
|
||||
|
Loading…
Reference in New Issue
Block a user