Otherwise, when building glibc and other packages, the "strip" from
bootstrapTools is used, which doesn't recognise some tags produced by
the newer "ld" from binutils.
I assume there's not much use for it during bootstrapping.
This fixes them as well, as curl was compiled against libnghttp2 but the
lib wasn't copied to the bootstrap tools.
Fixes#12632.
I think it's better to quote this variable in general, because it is
common and even documented to pass space-separated commands in there.
The greps should just fail in that case and `if` won't proceed
which seems fine for such cases, and it's certainly better than
passing additional unintended parameters to grep
(which was happening all the time before).
Doing it in an openssl setup hook only works if packages have openssl
as a build input - it doesn't work if they're using a program linked
against openssl.
Commit 6d928ab684 changed this to not
preserve timestamps. However, that results in non-determinism; in
particular, it gives us a broken $SOURCE_DATE_EPOCH (especially for
everything using fetchFromGitHub). Builds affected by timestamps <
1980 should be fixed in some other way (e.g. changing the timestamp to
some fixed date > 1980).
The ld-wrapper.sh script calls `readlink` in some circumstances. We need
to ensure that this is the `readlink` from the `coreutils` package so
that flag support is as expected.
This is accomplished by explicitly setting PATH at the top of each shell
script.
Without doing this, the following happens with a trivial `main.c`:
```
nix-env -f "<nixpkgs>" -iA pkgs.clang
$ clang main.c -L /nix/../nix/store/2ankvagznq062x1gifpxwkk7fp3xwy63-xnu-2422.115.4/Library -o a.out
readlink: illegal option -- f
usage: readlink [-n] [file ...]
```
The key element is the `..` in the path supplied to the linker via a
`-L` flag. With this patch, the above invocation works correctly on
darwin, whose native `/usr/bin/readlink` does not support the `-f` flag.
The explicit path also ensures that the `grep` called by `cc-wrapper.sh`
is the one from Nix.
Fixes#6447
This is used by some build tools to provide reproducible builds. See
https://reproducible-builds.org/specs/source-date-epoch/
for more info.
Later, we'll want to set this to a more intelligent value (such as the
most recent mtime of any source file).
So far if no configure script is found or no makefile,
the rest of the phase is skipped, *including* post-hooks.
I find that behavior unexpected/unintuitive.
Earlier version of this patch had problems due to me assuming
that $configureScript is always a simple path, but that turned out
to be false in many cases, e.g. perl.
This un-hardcodes the bootstrap tools passed into the Darwin stdenv and
thus allows us to quickly iterate on improving the design of the full
bootstrap process. We can easily change the contents of the bootstrap
tools and evaluate an entire bootstrap all the way up to real packages.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
You can now pass
separateDebugInfo = true;
to mkDerivation. This causes debug info to be separated from ELF
binaries and stored in the "debug" output. The advantage is that it
enables installing lean binaries, while still having the ability to
make sense of core dumps, etc.
Otherwise this fails on ARM:
/nix/store/jipqp9739n7wrjz40igbk85pqk13s0ad-binutils-2.23.1/bin/ld: /nix/store/92pdpqrqkdf8wjciq1cisvsp8kdz8p2i-gmp-5.1.3/lib/libgmp.a(mp_get_fns.o): relocation R_ARM_MOVW_ABS_NC against `__gmp_allocate_func' can not be used when making a shared object; recompile with -fPIC
/nix/store/92pdpqrqkdf8wjciq1cisvsp8kdz8p2i-gmp-5.1.3/lib/libgmp.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [libisl.la] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/tmp/nix-build-isl-0.11.1.drv-3/isl-0.11.1'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/nix-build-isl-0.11.1.drv-3/isl-0.11.1'
make: *** [all] Error 2
builder for ‘/nix/store/a8ghniifd8d8agqx0cqsh41daa08v11c-isl-0.11.1.drv’ failed with exit code 2
Several places in the tree associate the ARMv7 system
with the beaglebone platform. Change them to point to
armv7l-hf-multiplatform as it supports several boards (including the
beaglebone as well)
Now development stuff is propagated from the first output,
and userEnvPkgs from the one with binaries.
Also don't move *.la files (yet). It causes problems, and they're small.
- there were many easy merge conflicts
- cc-wrapper needed nontrivial changes
Many other problems might've been created by interaction of the branches,
but stdenv and a few other packages build fine now.
Attrnames and package names should be as close as possible to avoid confusion.
I took care not to confuse the two mpc things during the mass-replace,
so hopefully I suceeded (tarball still builds).
Conflicts (simple):
pkgs/os-specific/linux/util-linux/default.nix
It seems this merge creates a new stdenv hash,
because we had changes on both branches :-/
- IMO using a temporary is not needed here (anymore),
- temporary at that location can cause a problem (in a specific case):
for example, when using the substituteAll function from nixpkgs
on a single file directly under /nix/store/ (or ./foo-file),
the stdenv's substitute tries to create a temporary directly under
/nix/store, which causes problems on chrooted darwin
(according to @copumpkin earlier today on IRC)
The old boot.spl.hostid option was not working correctly due to an
upstream bug.
Instead, now we will create the /etc/hostid file so that all applications
(including the ZFS kernel modules, ZFS user-space applications and other
unrelated programs) pick-up the same system-wide host id. Note that glibc
(and by extension, the `hostid` program) also respect the host id configured in
/etc/hostid, if it exists.
The hostid option is now mandatory when using ZFS because otherwise, ZFS will
require you to force-import your ZFS pools if you want to use them, which is
undesirable because it disables some of the checks that ZFS does to make sure it
is safe to import a ZFS pool.
The /etc/hostid file must also exist when booting the initrd, before the SPL
kernel module is loaded, so that ZFS picks up the hostid correctly.
The complexity in creating the /etc/hostid file is due to having to
write the host ID as a 32-bit binary value, taking into account the
endianness of the machine, while using only shell commands and/or simple
utilities (to avoid exploding the size of the initrd).
Getting the names of all environment variables is tricky. The previous
implementation easily got confused by multi-line variables. The new
one is more reliable but not still not perfect.
This works around a segfault in Bash 4.3, where the expression
"${!var}" (where var="-9") crashes under certain conditions.
http://hydra.nixos.org/build/16693445
Otherwise, stdenv won't have a reference to e.g. patchelf on Linux
(because it was passed in by mkDerivation). This causes the installer
tests to fail, because having "stdenv" in the installation CD closure
is not enough to pull in all stdenv packages.
http://hydra.nixos.org/build/16546643
This allows licenses like the Amazon Software License to be identified
properly while still preventing packages with those licenses from
being distributed in the Nixpkgs/NixOS channels.
This makes stuff build with Xcode 6.1 on Mac OS X 10.9 (where we got
errors like "ld: file not found:
/usr/lib/system/libsystem_coreservices.dylib for architecture x86_64" due to the use of the 10.10 SDK).
The static curl program is gone, replaced by curl inside of the
bootstrap tools tarball. Also, we generate a .tar.xz archive rather
than .cpio.bz2, making the download smaller. The separate
{sh,cpio,mkdir,ln,bzip2} programs have been replaced by a single
busybox program.
In 3.3, a C++ class defined in a header will get a typeinfo symbol
like this (e.g. in Nix's src/libutil/util.o):
(__DATA,__datacoal_nt) weak external typeinfo for nix::BaseError
But in 3.4, this has changed to:
(__DATA,__datacoal_nt) weak external automatically hidden typeinfo for nix::BaseError
This causes the linker to change the symbol to:
(__DATA,__data) non-external (was signed char private external) typeinfo for nix::BaseError
i.e. losing its weak linkage. But without weak linkage, dynamic_cast
and other RTTI-based mechanisms (such as catching an exception of a
certain type) don't work across shared libraries / executables.
The clang compiler in the SDK doesn't have this behaviour, but it's
not clear exactly which version it is (it just says "based on LLVM
3.4svn").
This should fix the OpenJDK build, which was failing because paxctl is
in sbin and therefore not automatically added to $PATH.
http://hydra.nixos.org/build/15658346
Copying /usr/lib/system/libunwind.dylib at evaluation time doesn't
work (e.g. on Hydra). And copying binary system libraries is a bad
idea anyway for license reasons.
Make thread disabling explicit. This changes the semantics of the perl
derivation, so on other platforms it may require setting
enableThreading = false
This commit doesn't change the derivation or out hash of stdenvLinux.
The stage3.extraAttrs.glibc argument was required for this whole build
procedure to correctly work and it was very-very hard to see why (the
comment said something about gcc47, but we're using gcc48 now).
This stage3.extraAttrs.glibc goes into stage3.stdenv.glibc after some
arg passing, and in pkgs/development/compiler/gcc an (stdenv ? glibc)
boolean expression decides to override /usr/include during the GCC
build.
All of our stages are built with glibc, so this refactoring moves this
repeating specification of glibc (once for gcc wrapper and once here for
extraAttrs) to stageFun, by getting rid of wrapGCC, as we were using
that in all of the stages anyways.
Incidentally it turned out, that this stdenv.glibc inconsistency caused
some random other stuff to behave differently:
- stage1.pkgs.perl has threading disabled,
- stage4.pkgs.coreutils (the production coreutils) has testing disabled.
Leave this historical accidents as they are in this commit, so the scope
of this commit can stay as a refactoring only, these issues will be
fixed in separate commits.
This commit doesn't change the derivation or the output hash of
stdenvLinux.
Use the new allowedRequisites feature in stdenvLinux.
This way we properly check that the end-result stdenv of the quite
complicated multi-stage stdenvLinux building procedure is sane, and only
depends on the stuff that we know about.
Alternative would be to just disallowRequisites bootstrapTools, which is
the most common offender, but we have had other offenders in the past.
For these checks to actually fire, you currently have to use nixUnstable,
as the necessary feature will be released in Nix 1.8.
Previously stdenv depended on two different zlibs and there was a third
one in the top-level package set for other purposes. This commit merges
all this zlibs to one.
Actually this have been committed once as 1f2b636, but then got lost
while resolving merge conflicts. Hopefully it survives this time.
The point here is that it's always possible to debug this staging
logic by using `nix-repl pkgs/stdenv/linux'. The modular-stdenv
change introduced the lib dependency, which we now default to
../../../lib.
No derivation or out hashes of stdenvLinux is changed by this commit.
Previously stdenv depended on two different zlibs and there was a third
one in the top-level package set for other purposes. This commit merges
all this zlibs to one.
Now it should contain *all* information from stdenv/setup.sh of
the original mutiple-output branch.
However, the configurability of the output paths is much greater.
This is needed for multiple-output derivations,
where it is desirable to propagate deps and setup-hooks into $dev instead of $out.
Also drop an unused simple function which will not even make sense.
Make stages explicit and generalize the pattern of having an stdenv and
a pkgs collection for all stages to a common stage generating function
called stageFun.
Rewrite all stage handling with this new function.
This commit doesn't change the outhash (or drvhash) of the stdenv.
Don't use default parameter values, to make the callsites more readable
and for easier debuggability/changability. Also reordered the
callsites' parameter ordering for consistency.
In the final stdenv don't repeat the name of the shell.
This commit doesn't change the outhash (or drvhash) of the stdenv.
All the different stages of stdenv had the fetchurl inherited anyways,
so make this generic in stdenvBootFun.
This commit doesn't change the outhash (or drvhash) of the stdenv.
Now gcc is just another build input, making it possible in the future
to have a stdenv that doesn't depend on a C compiler. This is very
useful on NixOS, since it would allow trivial builders like
writeTextFile to work without pulling in the C compiler.
If $src refers to a directory, then always copy it. Previously, we
checked the extension first, so if the directory had an extension like
.tar, unpackPhase would fail.
If a build input is a regular file, use it as a setup hook. This makes
setup hooks more efficient to create: you don't need a derivation that
copies them to $out/nix-support/setup-hook, instead you can use the
file as is.
You can now register multiple values per named hook, e.g.
addHook preConfigure "echo foo"
addHook preConfigure "echo bar"
will cause ‘runHook preConfigure’ to run both ‘echo foo’ and ‘echo
bar’ (in that order). It will also call the shell function
preConfigure() or eval the shell variable $preConfigure, if
defined. Thus, if you don't call addHook, it works like the old hook
mechanism.
Allowing multiple hooks makes stdenv more modular and extensible. For
instance, multiple setup hooks can define a preFixup hook, and all of
these will be executed.
This variable sets the minimal Mac OS X version required for
running binaries produced by the Darwin toolchain. Since it
defaults to the version of the user's SDK, setting it explicitly
should make our builds more deterministic. It's now set to 10.6
because that's what hydra.nixos.org runs.
Commit 262c21ed46 purported to enable
ignoreNulls, but it was bogus because it set the flag on the wrong
derivation (i.e. stdenv rather than the result of mkDerivation).
Recent versions of Xcode don't install headers in /usr/include but
in a directory like
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include
So use that instead, falling back to /usr/include in case of an older
version of Xcode.
Commit 986f361946 started to use
<nix/fetchurl.nix> to "download" the bootstrap binaries from the
Nixpkgs tree, using the file:/// scheme. This has really bad
consequences:
* It makes any derivation depend on the path of the Nixpkgs tree. So
evaluating a package will produce a different .drv file when run
from different locations. No wonder Hydra evaluation has been so
slow lately: for every Nixpkgs evaluation, it had to create tens of
thousands of .drv files, even if nothing had changed.
* It requires the builder to have file system access to the Nixpkgs
tree. So if your tree is in your home directory, the stdenv
bootstrap would probably fail.
So now the binaries are downloaded from tarballs.nixos.org.
Also dropped PowerPC "support".
Stdenv adapters are kinda weird and un-idiomatic (especially when they
don't actually change stdenv). It's more idiomatic to say
buildInputs = [ makeCoverageAnalysisReport ];
This removes the need for hacks like stdenv.regenerate. It also
ensures that overrideGCC is now stackable (so ‘stdenv = useGoldLinker
clangStdenv’ works).
setup.sh uses the anti-pattern `for f in $(find ...); do` in several
places. `find` returns one path per line, but `for` splits its arguments
by words, so paths which contain spaces are incorrectly split! The
correct way is `find ... | while read f; do`
Binutils nowadays contains ld.gold, which depends on libstdc++. So it
needs to be built with the new GCC rather than the one from
bootstrap-tools.
Issue #1469.
Treating fixupPhase specially is really ugly. Also, it collides with
the work in the multiple-outputs branch (which already has support for
fixing up all outputs).
Partial revert of 0a44a09121.
Some programs, e.g. guile-config, has a shebang that ends in '\':
#!/usr/bin/guile-1.8 \
-e main -s
!#
;;;; guile-config --- utility for linking programs with Guile
;;;; Jim Blandy <jim@red-bean.com> --- September 1997
This currently breaks patchShebangs:
$ read oldPath arg0 args <<< 'shebang \'; echo $?
1
$ echo $oldPath
shebang
$ echo $arg0
$ echo $args
(And setup.sh/patchShebangs is run with 'set -e' so any command that
return non-zero aborts the build.)
Fix by telling 'read' to not interpret backslashes (with the -r flag):
$ read -r oldPath arg0 args <<< 'shebang \'; echo $?
0
$ echo $oldPath
shebang
$ echo $arg0
\
$ echo $args
Also needed: escape the escape characters so that sed doesn't interpret
them.
patchShebangs has a bug that shows itself on files that have the
executable bit set but have no shebang (i.e. a blank/empty first line).
The shell would then evaluate this:
if [ != '#!' ]; then
# not evaluated
fi
With proper quoting we get the correct behaviour:
if [ "" != '#!' ]; then
# this will be evaluated
fi
When building e.g. perl for the first time there is no perl in PATH yet,
so command -v perl will fail.
This brings back the previous behaviour of silently not patching
shebangs for which there is no available command in PATH.
Currently "/usr/bin/env python" is rewritten to "/nix/store/.../env
python". That doesn't really improve anything because the interpreter
still have to be located in $PATH at runtime. The result is that many
nix package expressions do .../bin/env fixup themselves.
Instead of everyone having to do this patching locally, add the
functionality to the standard environment patchShebangs function so that
everyone can benefit.
The function ‘mkDerivation’ now checks whether the current platform
type is included in a package's meta.platform field. If not, it
throws an exception:
$ nix-build -A linux --argstr system x86_64-darwin
error: user-thrown exception: the package ‘linux-3.10.15’ is not supported on ‘x86_64-darwin’
These packages also no longer show up in ‘nix-env -qa’ output. This
means, for instance, that the number of packages shown on
x86_64-freebsd has dropped from 9268 to 4764.
Since meta.platforms was also used to prevent Hydra from building some
packages, there now is a new attribute meta.hydraPlatforms listing the
platforms on which Hydra should build the package (which defaults to
meta.platforms).
meta.license is can be a string or a list of strings. But there is one
unhandled case where "unfree" (or "unfree-redistributable") is a part of
a list. It will currently not be detected as an "unfree" package and
Hydra will attempt to build it. This should fix it.
Example: http://hydra.nixos.org/build/6553461
GCC provides a number of libraries that are used by programs built by
GCC, in particular libgcc_s.so and libstdc++.so. This caused programs
that used these libraries to have a runtime dependency on all of GCC
(~77 MiB). Now they only depend on the "lib" output of GCC (~1.6
MiB).
With this and previous multiple-output improvements, closure sizes are
reduced a lot:
hello: 41 MiB -> 22 MiB
patchelf: 118 MiB -> 23 MiB
pan: 364 MiB -> 90 MiB
This reduces the size of closures a lot by getting rid of Glibc's
header files, static libraries and utilities. For instance, the
closure of GNU Hello went from 41 MiB to 22 MiB.
TODO: We may want to merge back in parts of Glibc that we previously
split off (such as the info files).
set CMAKE_LIBRARY_PATH, CMAKE_INCLUDE_PATH based on NIX_CFLAGS_COMPILE and
NIX_LDFLAGS so that cmake's find_library like functions find all the libraries
gcc knows about thanks to the gcc wrapper
This is particular useful with myEnvFun which then also sets those CMAKE_* env
variables.`
Because setup.sh has to change this causes many rebuilds - thus it should be
included in a stdenv-update like branch
Also cmake builds in parallel perfectly fine
update cmake to latest minor number, I didn't change the patches,
just reapplied them manually recordin a new patch
Since nix-1.4, nix's corepkgs contain a fetchurl suitable for
downloading the bootstrap binaries. Doing this will allow us to have a
nixpkgs with no in-tree binaries without breaking the purity of the
bootstrap (though for now, they are fetched in-tree until the binaries
are added to nixos.org somewhere). As an additional small benefit, the
in-tree binaries do not have to be hashed on every instantiation as they
do now.
The fetchurl in nix-1.2 and 1.3 isn't able to make binaries executable,
so it can't be used for this case. In that case, attempting to build the
bootstrap will show a message asking the user to manually download each
file and add it to the store with nix-store --add (but the hash is
ultimately the same, of course).
Signed-off-by: Shea Levy <shea@shealevy.com>
Conflict in kerberos, which was updated both in master and in
stdenv-updates. Kept the stdenv-updates version, except pulled in the
enableParallelBuilding change from master.
Signed-off-by: Shea Levy <shea@shealevy.com>
Conflicts:
pkgs/development/libraries/kerberos/krb5.nix
