We now have the information to properly determine the role the
cc-wrapper dependency has, by taking advantage of `offset`. No longer
use the soon-to-be-deprecated crossConfig environment variable, the
temp hack used before this change.
4 far-reaching changes: Smaller PATH, New vars, different propagation
logic, and different hook logic
Smaller PATH
------------
`buildInputs` no longer go on the PATH at build time, as they cannot be
run when cross compiling and we don't want to special case. Simply make
a `nativeBuildInput` too if one needs them on the PATH. Fixes#21191.
Many new depedendency variables
-------------------------------
See the stdenv chapter of the nixpkgs manual. I pulled out the existing
documentation of dependency specification into a new section, and added
language for these two (and their propagated equivalents) along side
the others'.
More complex propagation logic
------------------------------
Before a propagated*XXX*Input always acted as if it was specified
directly as a *XXX*Input downstream. That's simple enough, but violates
the intended roles of each sort of dep, which has functional and not
just stylistic consequences.
The new algorithm is detailed in the manual, and ensures everything
ends up in the right place. I tried to give both an informal and formal
description, but I suspect in practice it will not make much sense
until one tries cross compiling, after which it will immediately make
sense as the only sane option.
Simplified hook logic
---------------------
Rather than `envHook` and `crossEnvHook`, whose behavior differs
depending on whether we are cross compiling or not, there is now one
hook per sort (or rather non-propagated and propagated pair of sorts)
of dependency. These new hooks have the same meaning regardless of
cross compilation. See the setup hook section of stdenv chapter of the
Nixpkgs manual for more details.
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.
This is an incremental step to fully keeping the dependencies separate
in all cases.
I find the separation of concerns, accumulating, then processing, easier
to follow. Also, with my yet-to-be-merged cross work, the accumulation
part will become more complex.
Why 6? It seems a decently high number, giving us room for more degrees
of debugging before the `set -x` sledgehammer without incurring a
mass-rebuild.
Environment variable filter in substituteAll was not precise and produced
undefined and invalid variable names. Vladimír Čunát tried to fix that in [1],
but `env -0` did not work during Darwin bootstrap, so [2] reverted this change
and replaced an error due to invalid variables with a warning. Recently in #28057
John Ericson added `set -u` to `setup.sh` and undefined variables made the setup
fail during e.g. `nix-build -A gnat` with `setup: line 519: !varName: unbound
variable`.
[1] 62fc8859c1
[2] 81df035429
Older bash version, like those in the bootstrap tools and on macOS,
currently confuse variables defined as an empty array with undefined
variables. `${foo+"${foo[@]}"}` will prevent `set -u` problems with
empty arrays and older without making a single '' in the empty case.
Care is taken to `set +u` when running hooks so as to not break existing
packages.
This reverts commit eeabf85780.
This change suddenly makes tons of stdenv internals visible in
nativeBuildInputs of every derivation, which doesn't seem desirable.
E.g:
````
nix-repl> hello.nativeBuildInputs
[ «derivation /nix/store/bcfkyf6bhssxd2vzwgzmsbn7b5b9rpxc-patchelf-0.9.drv»
«derivation /nix/store/4wnshnz9wwanpfzcrdd76rri7pyqn9sk-paxctl-0.9.drv»
<< snip 10+ lines >>
«derivation /nix/store/d35pgh1lcg5nm0x28d899pxj30b8c9b2-gcc-wrapper-6.4.0.drv»
]
````
Additionally, instead of pulling them from `setup.sh`, route them via
Nix. This gets us one step closer to making stdenv be a plain attribute
set instead of a derivation.
@vcunat and others rightly point out that it's easier to quote always,
than learn Bash's idiosyncrasies enough to know when it doesn't make a
difference.
This reverts commit 2743078f66, which
removes quotes that don't do anything, and then goes further adding
even more quotes.
I took some liberties with the flags-echoing code to make it more
concise and correct. Also, a few warnings in findInputs and friends I
skipped because I am going to rewrite those anyways.
Thanks @grahamc for telling me about this great linter!
This makes those files a bit easier to read. Also, for what it's worth,
it brings us one baby step closer to handling spaces in store paths.
Also, I optimized handling of many transitive deps with read. Probably,
not very beneficial, but nice to enforce the pkg-per-line structure.
Doing so let me find much dubious code and fix it.
Two misc notes:
- `propagated-user-env-packages` also needed to be adjusted as
sometimes it is copied to/from the propagated input files.
- `local fd` should ensure that file descriptors aren't clobbered
during recursion.
When not cross compiling, nativeBuildInputs and buildInputs have
identical behaviour. Currently that is implemented by having
mkDerivation do a concatenation of those variables in Nix code and pass
that to the builder via the nativeBuildInputs attribute.
However, that has some annoying side effects, like `foo.buildInputs`
evaluating to `[ ]` even if buildInputs were specified in the nix
expression for foo.
Instead, pass buildInputs and nativeBuildInputs in separate variables as
usual, and move the logic of cross compilation vs. native compilation to
the stdenv builder script. This is probably a tiny bit uglier but
fixes the previous problem.
Issue #4855.
`stripHash` documentation states that it prints out the stripped name to
the stdout, but the function stored the value in `strippedName`
instead.
Basically all usages did something like
`$(stripHash $foo | echo $strippedName)` which is just braindamaged.
Fixed the implementation and all invocations.
Close#15803. This avoids the error:
while setting up the build environment: executing
‘/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin/bash’:
Argument list too long
Note: I wanted to make it optional based on buildCommand length,
but that seems pointless as I'm sure it's less performant.
Amended by vcunat:
https://github.com/NixOS/nixpkgs/pull/15803#issuecomment-224841225
On Linux, paxctl's setup hook should overwrite the paxmark stub, but the
stub is defined after the setup hooks are sourced, so the stub ends up
overwriting the real function. The result is that paxmark fails to do
anything. The fix is to define the stub before any setup hooks are
sourced. Thanks to @vcunat for figuring this out.
Closes#15492
I'm giving this up. Feel free to find some reasonable variant that works
at least on Linux and Darwin. Problems encountered:
- During bootstrap of Darwin stdenv `env -0` and some bash features
don't work.
- Without `env -0` the contents of some multi-line phases is taken as
variable declarations, which wouldn't typically matter, but the PR
wanted to refuse bash-invalid names which would be occasionally
triggered. This commit dowgrades that to a warning with explanation.
It turned out that process substitution fed into a while-cycle
isn't recognized during darwin bootstrap:
http://hydra.nixos.org/build/35382446/nixlog/1/raw
Also fix broken NIX_DEBUG output, noticed by abbradar.
The set/env fix in #14907 wasn't very good, so let's use a null-delimited
approach. Suggested by Aszlig.
In particular, this should fix a mass-breakage on Darwin, though I was
unable to test that.
bash variable names may only contain alphanumeric ASCII-symbols and _,
and must not start with a number. Nix expression attribute names however
might contain nearly every character (in particular spaces and dashes).
Previously, a substitution that was not a valid bash name would be
expanded to an empty string. This commit introduce a check that throws
a (hopefully) helpful error when a wrong name is used in a substitution.
Close#14335.
Since 89036ef76a, when a package doesn't include a configure script,
the build complains with:
grep: : No such file or directory
grep: : No such file or directory
This prevents that.
Otherwise, when building glibc and other packages, the "strip" from
bootstrapTools is used, which doesn't recognise some tags produced by
the newer "ld" from binutils.
Fixes#12632.
I think it's better to quote this variable in general, because it is
common and even documented to pass space-separated commands in there.
The greps should just fail in that case and `if` won't proceed
which seems fine for such cases, and it's certainly better than
passing additional unintended parameters to grep
(which was happening all the time before).
Doing it in an openssl setup hook only works if packages have openssl
as a build input - it doesn't work if they're using a program linked
against openssl.
Commit 6d928ab684 changed this to not
preserve timestamps. However, that results in non-determinism; in
particular, it gives us a broken $SOURCE_DATE_EPOCH (especially for
everything using fetchFromGitHub). Builds affected by timestamps <
1980 should be fixed in some other way (e.g. changing the timestamp to
some fixed date > 1980).
This is used by some build tools to provide reproducible builds. See
https://reproducible-builds.org/specs/source-date-epoch/
for more info.
Later, we'll want to set this to a more intelligent value (such as the
most recent mtime of any source file).
So far if no configure script is found or no makefile,
the rest of the phase is skipped, *including* post-hooks.
I find that behavior unexpected/unintuitive.
Earlier version of this patch had problems due to me assuming
that $configureScript is always a simple path, but that turned out
to be false in many cases, e.g. perl.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
Now development stuff is propagated from the first output,
and userEnvPkgs from the one with binaries.
Also don't move *.la files (yet). It causes problems, and they're small.
- there were many easy merge conflicts
- cc-wrapper needed nontrivial changes
Many other problems might've been created by interaction of the branches,
but stdenv and a few other packages build fine now.
- IMO using a temporary is not needed here (anymore),
- temporary at that location can cause a problem (in a specific case):
for example, when using the substituteAll function from nixpkgs
on a single file directly under /nix/store/ (or ./foo-file),
the stdenv's substitute tries to create a temporary directly under
/nix/store, which causes problems on chrooted darwin
(according to @copumpkin earlier today on IRC)
Getting the names of all environment variables is tricky. The previous
implementation easily got confused by multi-line variables. The new
one is more reliable but not still not perfect.
This works around a segfault in Bash 4.3, where the expression
"${!var}" (where var="-9") crashes under certain conditions.
http://hydra.nixos.org/build/16693445
Otherwise, stdenv won't have a reference to e.g. patchelf on Linux
(because it was passed in by mkDerivation). This causes the installer
tests to fail, because having "stdenv" in the installation CD closure
is not enough to pull in all stdenv packages.
http://hydra.nixos.org/build/16546643
Now it should contain *all* information from stdenv/setup.sh of
the original mutiple-output branch.
However, the configurability of the output paths is much greater.
This is needed for multiple-output derivations,
where it is desirable to propagate deps and setup-hooks into $dev instead of $out.
Also drop an unused simple function which will not even make sense.
Now gcc is just another build input, making it possible in the future
to have a stdenv that doesn't depend on a C compiler. This is very
useful on NixOS, since it would allow trivial builders like
writeTextFile to work without pulling in the C compiler.
If $src refers to a directory, then always copy it. Previously, we
checked the extension first, so if the directory had an extension like
.tar, unpackPhase would fail.
If a build input is a regular file, use it as a setup hook. This makes
setup hooks more efficient to create: you don't need a derivation that
copies them to $out/nix-support/setup-hook, instead you can use the
file as is.
You can now register multiple values per named hook, e.g.
addHook preConfigure "echo foo"
addHook preConfigure "echo bar"
will cause ‘runHook preConfigure’ to run both ‘echo foo’ and ‘echo
bar’ (in that order). It will also call the shell function
preConfigure() or eval the shell variable $preConfigure, if
defined. Thus, if you don't call addHook, it works like the old hook
mechanism.
Allowing multiple hooks makes stdenv more modular and extensible. For
instance, multiple setup hooks can define a preFixup hook, and all of
these will be executed.
