musl now supports RISC-V. Let's centralise musl availability checks
in musl.meta.platforms, so we don't have to keep cleaning up ad-hoc
checks like this all over the tree.
I broke `pkgsMusl` with #209870.
Something odd is happening with `xgcc` (the temporary compiler that
should be used only to compile `gcc`, although we are using it to
compile a temporary `patchelf` too) and `libstdc++`.
The temporary fix in this commit is to use `-static-libstdc++` for
the ephemeral `patchelf` built by `xgcc`. It isn't pretty, but it
appears to work.
Incorporates:
- https://github.com/NixOS/nixpkgs/pull/224945
The stage before `xgcc` creates the first compiled patchelf
(i.e. not from bootstrapFiles).
The `xgcc` stage was inadvertently switching *back* to using the
patchelf *from* the bootstrapFiles.
The first commit in this PR adds self-checking comments (assertions)
to make it clear where each stage's patchelf comes from.
The second commit fixes the bug, and updates the self-checking
comments.
Without the change when I attempt to built `nixpkgs` with weekly
`gcc-13` (it pulls in `flex` as a build input`) I am getting build
failure related to glibc mix caused by glibc loading:
...-binutils-patchelfed-ld-2.40/bin/ld: ...-xgcc-13.0.0/libexec/gcc/x86_64-unknown-linux-gnu/13.0.1/liblto_plugin.so:
error loading plugin: ...-bootstrap-tools/lib/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE
The change disables LTO plugin entirely to avoid loading of `glibc` mix.
This commit adds `gcc/common/checksum.nix`, which contains code
common to both gcc11 and gcc12, implementing the `enableChecksum`
feature.
When gcc's built-in bootstrap (`--enable-bootstrap`) is used, gcc
compiles itself three times and compares a hash of the unlinked `.o`
files from the second and third compilation. The
`enableChecksum=true` parameter performs the same comparison as part
of the `postInstall` phase.
Notably, `enableChecksum=true` can be used with `enableBootstrap=false`.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
Our bootstrap-files unpacker has always relied on a lot of unstated
assumptions, one of them being that every library has a DT_NEEDED
for librt.so, so patchelf'ing something into the RUNPATH into
librt.so means that it will be searched for every library load in
all of the bootstrap-files.
Unfortunately that assumption is not true for libgcc.
This causes problems, because patchelf links against libgcc (and
against libstdc++, which links against libgcc). So we can't use
patchelf on libgcc, because it needs libgcc, so patchelf doesn't
work until libgcc is patchelfed.
The robust solution here is to use static linking for the copy of
patchelf that is shipped with the bootstrap-files. We don't have to
go all the way to a statically linked libc; just -static-libgcc and
-static-libstdc++ are enough to break the circular dependency.
Right now our bootstrapFiles-selecting algorithm uses the
`loongson2f.nix` bootstrapFiles (which were not built by Hydra).
These bootstrapFiles don't work anymore. They were added in 2010 by
40405d03ac.
This commit causes mipsel-linux native builds to use the Hydra-built
bootstrap files from this PR instead:
https://github.com/NixOS/nixpkgs/pull/183487
Hydra job building them: https://hydra.nixos.org/build/208909151
The bootstrap files can be reproduced on the commit 21ec906463, e.g. by:
cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=
You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
PR #208478 added a lot of documentation about which packages were
rebuilt in each stage of the stdenv bootstrap. However nothing
checks that these comments agree with reality; they can bitrot over
time. This PR rewrites those comments as assertions, so they cannot
bitrot.
This conversion did expose some ambiguity in our scheme for naming
the stages. Suppose that `pkgs.stdenv.name=="stdenv-stage4", then
which of these is "the stage4 coreutils"?
```
pkgs.coreutils
pkgs.stdenv.__bootPackages.coreutils
```
The choice is arbitrary, and both choices have confusing corner
cases. We should revisit this at some point.
Hydra job building them: https://hydra.nixos.org/build/208909151
The bootstrap files can be reproduced on the parent commit, e.g. by:
cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=
You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
`builtins.baseNameOf` retains any string context, causing the test
derivation to incorrectly depend on `pkgs.glibc`. All we really want is
to know what the dynamicLinker is called, but we don't need it to be
present in store.
Thanks to Adam Joseph for spotting this.
After https://github.com/NixOS/nixpkgs/pull/209054 we started moving
libstdc++.so out of default glibc's paths. This exposed bootstrap
tools build failure as:
$ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
...
>
.../bin/bar: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory
Note that bootstrap itself did not break. The change only expands
handcrafted `-rpath` entries.
The stdenv bootstrap creates several wrappers around binutils, and
gives them the exact same drvName as the binutils package itself.
These wrappers cost almost nothing to create (they are just file
copies and patchelf runs, not builds), so we should distinguish them
from expensive binutils builds with a unique pname. This commit
does that.
Right now we build gettext several times during the bootstrap.
Gettext's build process is "embarrassingly serial", so avoiding
rebuilding it speeds things up considerably.
This change allows building new gcc during bootstrap without fear of
pulling in outdated libstdc++.so after g++ switched from bootstrapTools
to freshly built g++.
Noticed when tried to add early bootstrap stage to rebuild `gcc` before
`glibc` is fully untangled from `bootstrapTools` as a failure to built
`binutils`:
ld: dwp.o: in function `__gnu_cxx::new_allocator<gold::Dwp_output_file::Contribution>::allocate(unsigned long, void const*)':
/nix/store/...-gcc-11.3.0/include/c++/11.3.0/ext/new_allocator.h:116: undefined reference to `std::__throw_bad_array_new_length()'
The change survives existing bootstrap and unblockes early `gcc` bootstrap.
Just a few comments added:
- added a few one-liners to explore which tools are rebuilt at each
stdenv iteration during bootstrap
- explicitly listed available toolchains and their sources for on each
bootstrap step: glibc, binutils, gcc, coreutils.
- added mention of static libraries linked into gcc
Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
This PR provides the Hydra-generated bootstrap tarballs for mips64el-linux-gnuabin32.
With this PR we now have the bootstrap-file for all three little-endian ABIs on mips: n64, n32, and o32. I do not currently plan to do big-endian mips unless some motivation arises; all mips chips are bi-endian and Debian has dropped big-endian support due to lack of interest.
I'll be following the script used in #151399, #168199, and #183487.
Files came from [this](https://hydra.nixos.org/build/188389586#tabs-summary) Hydra build, which used nixpkgs revision 97d9c84e1d to instantiate:
```
/nix/store/hakn8s85s9011v61r6svp5qy8x1y64fv-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32.drv
```
and then built:
```
/nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
```
I downloaded these files from Hydra and prefetched them into the nix store with the following commands:
```
STOREPATH=rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store prefetch-file \
file://$(nix store add-file --name bootstrap-tools.tar.xz $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix store prefetch-file --executable \
file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
```
These commands produced the following output:
```
Downloaded 'file:///nix/store/w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' to '/nix/store/6w0f0mqblrghvh6yjwcb4xdqq9x50lbl-w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' (hash 'sha256-LWrpN6su2yNVurUyhZP34OiZyzgh7MfN13fIIbou8KI=').
Downloaded 'file:///nix/store/nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' to '/nix/store/i361xhbdhhnvg7zd637xpm63vbl80s0s-nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' (hash 'sha256-4N3G1qYA7vitjhsIW17pR6UixIuzrq4vZXa8F0/X4iI=').
```
I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit.
I then started the bootstrap with the following command:
```
nix build -L -f . --arg localSystem '(import ./lib).systems.examples.mips64el-linux-gnuabin32' hello
```
As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
sha256sum /nix/store/${STOREPATH}/on-server/*
```
which produced the following output:
```
2d6ae937ab2edb2355bab5328593f7e0e899cb3821ecc7cdd777c821ba2ef0a2 /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/bootstrap-tools.tar.xz
01633f71135cb9ab1b5ce3ebb67e80cbf288739729bffc1350c1552f6f8df34b /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/busybox
```
gcc stopped using libelf in commit 48215350c24 ("re PR lto/46273 (Failed
to bootstrap)") around 2010, before gcc-4.6.0.
Bootstrap tools don't use it either.
This PR updates the Hydra-generated bootstrap tarballs for
powerpc64le-linux. The bootstrap-files referenced prior to this
commit will only bootstrap in a nixpkgs which has
https://github.com/NixOS/nixpkgs/pull/181802. That PR was closed in
favor of https://github.com/NixOS/nixpkgs/pull/181943, which is a
cleaner solution but which requires regenerating the bootstrap-files.
I'll be following the script established in
https://github.com/NixOS/nixpkgs/pull/151399, which I previously used
in https://github.com/NixOS/nixpkgs/pull/168199.
Files came from [this](https://hydra.nixos.org/build/186237511) Hydra build, which used nixpkgs revision ac43c44478 to instantiate:
```
/nix/store/nhjbza9vlcyhp9zxfz6lwpc3m2ghrpzj-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
```
and then built:
```
/nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
```
I downloaded these files from Hydra with the following commands:
```
STOREPATH=fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
I then prefetched them into `/nix/store` with:
```
$ nix store prefetch-file --executable file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox
Downloaded 'file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' to '/nix/store/a42qf2kf5hychcsw5sz0pvghy9vli1im-p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' (hash 'sha256-jtPEAsht4AUAG4MLK8xocQSfveUR4ppU1lS4bGI1VN4=').
$ nix store prefetch-file file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' to '/nix/store/kgzyq9q08nll28ccqjcbv8angq5hyvdp-y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' (hash 'sha256-MpIDnpZUK3M17qlnuoxfnK0EgxRosm3TMW1WfPZ1+jU=').
```
And started the bootstrap with the following command:
```
nix build -f . -L hello
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
3292039e96542b7335eea967ba8c5f9cad04831468b26dd3316d567cf675fa35 /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
3d078dff7b4087d82442937667c91dace3321493aae4d3a4160d046b7eabcc2c /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
