Go to file
adisbladis d88a7735d2
Python: introduce NIX_PYTHONPREFIX in order to set site.PREFIXES
This is needed in case of `python.buildEnv` to make sure site.PREFIXES
does not only point to the unwrapped executable prefix.

--------------------------------------------------------------------------------

This PR is a story where your valiant hero sets out on a very simple adventure but ends up having to slay dragons, starts questioning his own sanity and finally manages to gain enough knowledge to slay the evil dragon and finally win the proverbial price.

It all started out on sunny spring day with trying to tackle the Nixops plugin infrastructure and make that nice enough to work with.

Our story begins in the shanty town of [NixOps-AWS](https://github.com/nixos/nixops-aws) where [mypy](http://mypy-lang.org/) type checking has not yet been seen.

As our deuteragonist (@grahamc) has made great strides in the capital city of [NixOps](https://github.com/nixos/nixops) our hero wanted to bring this out into the land and let the people rejoice in reliability and a wonderful development experience.

The plugin work itself was straight forward and our hero quickly slayed the first small dragon, at this point things felt good and our hero thought he was going to reach the town of NixOps-AWS very quickly.

But alas! Mypy did not want to go, it said:
`Cannot find implementation or library stub for module named 'nixops'`

Our hero felt a small sliver of life escape from his body. Things were not going to be so easy.

After some frustration our hero discovered there was a [rule of the land of Python](https://www.python.org/dev/peps/pep-0561/) that governed the import of types into the kingdom, more specificaly a very special document (file) called `py.typed`.
Things were looking good.

But no, what the law said did not seem to match reality. How could things be so?

After some frustrating debugging our valiant hero thought to himself "Hmm, I wonder if this is simply a Nix idiosyncrasy", and it turns out indeed it was.
Things that were working in the blessed way of the land of Python (inside a `virtualenv`) were not working the way they were from his home town of Nix (`nix-shell` + `python.withPackages`).

After even more frustrating attempts at reading the mypy documentation and trying to understand how things were supposed to work our hero started questioning his sanity.
This is where things started to get truly interesting.

Our hero started to use a number of powerful weapons, both forged in the land of Python (pdb) & by the mages of UNIX (printf-style-debugging & strace).

After first trying to slay the dragon simply by `strace` and a keen eye our hero did not spot any weak points.
Time to break out a more powerful sword (`pdb`) which also did not divulge any secrets about what was wrong.

Our hero went back to the `strace` output and after a fair bit of thought and analysis a pattern started to emerge. Mypy was looking in the wrong place (i.e. not in in the environment created by `python.withPackages` but in the interpreter store path) and our princess was in another castle!

Our hero went to the pub full of old grumpy men giving out the inner workings of the open source universe (Github) and acquired a copy of Mypy.
He littered the code with print statements & break points.
After a fierce battle full of blood, sweat & tears he ended up in 20f7f2dd71/mypy/sitepkgs.py and realised that everything came down to the Python `site` module and more specifically https://docs.python.org/3.7/library/site.html#site.getsitepackages which in turn relies on https://docs.python.org/3.7/library/site.html#site.PREFIXES .

Our hero created a copy of the environment created by `python.withPackages` and manually modified it to confirm his findings, and it turned out it was indeed the case.
Our hero had damaged the dragon and it was time for a celebration.

He went out and acquired some mead which he ingested while he typed up his story and waited for the dragon to finally die (the commit caused a mass-rebuild, I had to wait for my repro).

In the end all was good in [NixOps-AWS](https://github.com/nixos/nixops-aws)-town and type checks could run. (PR for that incoming tomorrow).
2020-03-14 21:39:31 +00:00
.github .github/CODEOWNERS: remove myself from the Haskell code owners 2020-02-21 10:01:41 +01:00
doc doc: tiny grammar improvement in the same sentence again 2020-03-10 10:08:26 +01:00
lib Merge pull request #70157 from teto/lib_kernel 2020-03-12 23:53:42 +01:00
maintainers maintainers: update mmlb email 2020-03-14 03:36:01 +00:00
nixos Merge pull request #82468 from Mic92/kvmgt 2020-03-14 07:17:28 +00:00
pkgs Python: introduce NIX_PYTHONPREFIX in order to set site.PREFIXES 2020-03-14 21:39:31 +00:00
.editorconfig Revert ".version: remove final newline" 2018-04-28 14:23:13 +02:00
.gitattributes gitattributes: disable merge=union in all-packages 2018-03-27 11:03:03 -05:00
.gitignore Replace androidenv by new implementation 2018-12-18 21:16:06 +01:00
.version 20.09 is Nightingale 2020-02-10 14:14:18 -05:00
COPYING COPYING: include 2020 2020-01-11 15:17:22 -08:00
default.nix Fix local path to release notes in error message 2018-10-08 05:43:15 -05:00
flake.nix flake.nix: Add note 2020-02-10 16:36:53 +01:00
README.md readme: add link to community chat options 2020-02-11 23:25:10 -08:00

NixOS logo

Code Triagers badge Open Collective supporters

Nixpkgs is a collection of over 40,000 software packages that can be installed with the Nix package manager. It also implements NixOS, a purely-functional Linux distribution.

Manuals

  • NixOS Manual - how to install, configure, and maintain a purely-functional Linux distribution
  • Nixpkgs Manual - contributing to Nixpkgs and using programming-language-specific Nix expressions
  • Nix Package Manager Manual - how to write Nix expressions (programs), and how to use Nix command line tools

Community

Other Project Repositories

The sources of all official Nix-related projects are in the NixOS organization on GitHub. Here are some of the main ones:

Continuous Integration and Distribution

Nixpkgs and NixOS are built and tested by our continuous integration system, Hydra.

Artifacts successfully built with Hydra are published to cache at https://cache.nixos.org/. When successful build and test criteria are met, the Nixpkgs expressions are distributed via Nix channels.

Contributing

Nixpkgs is among the most active projects on GitHub. While thousands of open issues and pull requests might seem a lot at first, it helps consider it in the context of the scope of the project. Nixpkgs describes how to build over 40,000 pieces of software and implements a Linux distribution. The GitHub Insights page gives a sense of the project activity.

Community contributions are always welcome through GitHub Issues and Pull Requests. When pull requests are made, our tooling automation bot, OfBorg will perform various checks to help ensure expression quality.

The Nixpkgs maintainers are people who have assigned themselves to maintain specific individual packages. We encourage people who care about a package to assign themselves as a maintainer. When a pull request is made against a package, OfBorg will notify the appropriate maintainer(s). The Nixpkgs committers are people who have been given permission to merge.

Most contributions are based on and merged into these branches:

  • master is the main branch where all small contributions go
  • staging is branched from master, changes that have a big impact on Hydra builds go to this branch
  • staging-next is branched from staging and only fixes to stabilize and security fixes with a big impact on Hydra builds should be contributed to this branch. This branch is merged into master when deemed of sufficiently high quality

For more information about contributing to the project, please visit the contributing page.

Donations

The infrastructure for NixOS and related projects is maintained by a nonprofit organization, the NixOS Foundation. To ensure the continuity and expansion of the NixOS infrastructure, we are looking for donations to our organization.

You can donate to the NixOS foundation by using Open Collective:

License

Nixpkgs is licensed under the MIT License.

Note: MIT license does not apply to the packages built by Nixpkgs, merely to the files in this repository (the Nix expressions, build scripts, NixOS modules, etc.). It also might not apply to patches included in Nixpkgs, which may be derivative works of the packages to which they apply. The aforementioned artifacts are all covered by the licenses of the respective packages.