It may seem nice and abstract to just override the default version, but
that breaks the alias relationship where the original llvmPackages_* is
no longer in sync. Put another away, modifying the referee rather
instead of breaking the reference "copy-on-write" is impossible.
We want `buildPackages` to be almost the same as
`buildPackages.buildPackges`, but that is only true if most packages
don't care about the target platform. The commented code however made
them all care about whether the target platform was Darwin.
The hack of using `crossConfig` to enforce stricter handling of
dependencies is replaced with a dedicated `strictDeps` for that purpose.
(Experience has shown that my punning was a terrible idea that made more
difficult and embarrising to teach teach.)
Now that is is clear, a few packages now use `strictDeps`, to fix
various bugs:
- bintools-wrapper and cc-wrapper
Note that a bunch of non-python packages use this attribute already.
Some of those are clearly unaware of the fact that this attribute does
not exists in stdenv because they define it but don't to add it to
their `bulidInputs` :)
Also note that I use `buildInputs` here and only handle regular
builds because python and haskell builders do it this way and I'm not
sure how to properly handle the cross-compilation case.
As in:
$ nix eval -f . bash
Also remove the glibc propagation inherit that made these necessary,
stages handle propagating libc themselves (apparently) and
AFAICT no hashes are changed as a result of this.
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
(cherry picked from commit ba52ae5048)
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
This allows one to force a compiler to use native machine optimizations. This
goes contrary to all the usual guarantees of Nix and so should be used only by
end-user and only in specific cases when they know what are they doing.
In my case this is needed to get a noticeable FPS boost in RPCS3 which is very
CPU-hungry PlayStation 3 emulator.
- `localSystem` is added, it strictly supercedes system
- `crossSystem`'s description mentions `localSystem` (and vice versa).
- No more weird special casing I don't even understand
TEMP
Since at least d7bddc27b2, we've had a
situation where one should depend on:
- `stdenv.cc.bintools`: for executables at build time
- `libbfd` or `libiberty`: for those libraries
- `targetPackages.cc.bintools`: for exectuables at *run* time
- `binutils`: only for specifically GNU Binutils's executables,
regardless of the host platform, at run time.
and that commit cleaned up this usage to reflect that. This PR flips the
switch so that:
- `binutils` is indeed unconditionally GNU Binutils
- `binutils-raw`, which previously served that role, is gone.
so that the correct usage will be enforced going forward and everything
is simple.
N.B. In a few cases `binutils-unwrapped` (which before and now was
unconditionally actual GNU binutils), rather than `binutils` was used to
replace old `binutils-raw` as it is friendly towards some cross
compilation usage by avoiding a reference to the next bootstrapping
change.
First, we need check against the host platform, not the build platform.
That's simple enough.
Second, we move away from exahustive finite case analysis (i.e.
exhaustively listing all platforms the package builds on). That only
work in a closed-world setting, where we know all platforms we might
build one. But with cross compilation, we may be building for arbitrary
platforms, So we need fancier filters. This is the closed world to open
world change.
The solution is instead of having a list of systems (strings in the form
"foo-bar"), we have a list of of systems or "patterns", i.e. attributes
that partially match the output of the parsers in `lib.systems.parse`.
The "check meta" logic treats the systems strings as an exact whitelist
just as before, but treats the patterns as a fuzzy whitelist,
intersecting the actual `hostPlatform` with the pattern and then
checking for equality. (This is done using `matchAttrs`).
The default convenience lists for `meta.platforms` are now changed to be
lists of patterns (usually a single pattern) in
`lib/systems/for-meta.nix` for maximum flexibility under this new
system.
Fixes#30902
Resolved the following conflicts (by carefully applying patches from the both
branches since the fork point):
pkgs/development/libraries/epoxy/default.nix
pkgs/development/libraries/gtk+/3.x.nix
pkgs/development/python-modules/asgiref/default.nix
pkgs/development/python-modules/daphne/default.nix
pkgs/os-specific/linux/systemd/default.nix
We go out of our way (see top of file) to build a single binary
with symlinks for all of the tools, but were losing them
when preparing the bootstrap tools.
For the cc of the intermediate stages, to be precise. Doing the same for
bintools requires lots of refactoring.
This is mainly for the future extensibility as now you can change
documentation generation with impunity without rebuilding the
whole of stdenv.
Existing "mips64el" should be "mipsel".
This is just the barest minimum so that nixpkgs can recognize them as
systems - although required for building individual derivations onto
MIPS boards, it is not sufficient if you want to actually build nixos on
those targets
Aarch64 tools tested briefly with qemu-aarch64,
but neither have been actually used yet :).
For now only "host" indirectly via binary cache
at cache.allvm.org.
This is a temporary workaround to make `nix-env -qa` and `nix search` ignore
broken packages as they they did before this patchset.
This patch should be reverted after `nix` gets a proper fix for this.
See NixOS/nix#1771.
This option makes `meta.evaluate` into a close approximation of the result of
evaluating `.outPath` by checking all the dependencies recursively at a cost of
2x slowdown. Note that actually evaluating `.outPath` costs some
5x-7x more because `.outPath` also computes all the hashes.
I hope this will be a temporary measure. If there is consensus around
issue #33599, then we can follow an explicit `dontCheck`, but default to
not checking during cross builds when none is given.
This pushes check-meta evaluation to derivation evaluation step, leaving all other
attributes accessible.
Before this commit:
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
as expected
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
which is unfortunate since its impossible to use packages in autogenerated
documentation on all platforms.
After this commit:
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux
still fails
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> "xen-4.5.5"
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> "Xen hypervisor and related components (vanilla)"
- All deps go on the PATH
- CC and Bintools wrappers with their host != depender's host still get their
setup hooks run.
- Environment hooks get applied to all packages
This isn't so elegent, but eases the transition on a very significant
PR.
We now have the information to properly determine the role the
cc-wrapper dependency has, by taking advantage of `offset`. No longer
use the soon-to-be-deprecated crossConfig environment variable, the
temp hack used before this change.
4 far-reaching changes: Smaller PATH, New vars, different propagation
logic, and different hook logic
Smaller PATH
------------
`buildInputs` no longer go on the PATH at build time, as they cannot be
run when cross compiling and we don't want to special case. Simply make
a `nativeBuildInput` too if one needs them on the PATH. Fixes#21191.
Many new depedendency variables
-------------------------------
See the stdenv chapter of the nixpkgs manual. I pulled out the existing
documentation of dependency specification into a new section, and added
language for these two (and their propagated equivalents) along side
the others'.
More complex propagation logic
------------------------------
Before a propagated*XXX*Input always acted as if it was specified
directly as a *XXX*Input downstream. That's simple enough, but violates
the intended roles of each sort of dep, which has functional and not
just stylistic consequences.
The new algorithm is detailed in the manual, and ensures everything
ends up in the right place. I tried to give both an informal and formal
description, but I suspect in practice it will not make much sense
until one tries cross compiling, after which it will immediately make
sense as the only sane option.
Simplified hook logic
---------------------
Rather than `envHook` and `crossEnvHook`, whose behavior differs
depending on whether we are cross compiling or not, there is now one
hook per sort (or rather non-propagated and propagated pair of sorts)
of dependency. These new hooks have the same meaning regardless of
cross compilation. See the setup hook section of stdenv chapter of the
Nixpkgs manual for more details.
stdenvNoCC should not inject any C++ standard library, just as it
doesn't inject any C standard library. stdenv still does, but only
indirectly through stdenv.cc. Wrapped clangs can be simplified now that
they don't need to worry about clobbering CoreFoundation when replacing
the C++ standard library implementation.
This generally-good cleanup should assist with debugging some C++
failures in #26805.
- tracing seems annoying enough
- we get errors for all packages instead of aborting on the first one
- easier to differentiate from unwanted packages (broken, unfree, etc.)
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.
This is an incremental step to fully keeping the dependencies separate
in all cases.
I find the separation of concerns, accumulating, then processing, easier
to follow. Also, with my yet-to-be-merged cross work, the accumulation
part will become more complex.
One should do this when needed executables at run time. It is more
honest and cross-friendly than refering to binutils directly, if one
neeeds the default binary tools for the target platform, rather than
binutils in particular.
This requires some small changes in the stdenv, then working around the
weird choice LLVM made to hardcode @rpath in its install name, and then
lets us remove a ton of annoying workaround hacks in many of our Go
packages. With any luck this will mean less hackery going forward.
cc-wrapper may wrap a cc-compiler, but it doesn't need one to build
itself. (c.f. expand-response-params is a separate derivation.) This
helps avoid cycles on the cross stuff, in addition to removing a
useless dependency edge.
I could have been super careful with overrides in the stdenv to avoid
the mass rebuild, but I don't think it's worth it.
Why 6? It seems a decently high number, giving us room for more degrees
of debugging before the `set -x` sledgehammer without incurring a
mass-rebuild.
By default, all previous overrides are discarded as before, as they
would only apply to the old host platform. But sometimes it is useful to
add some new ones, and this optional parameter allows that.
This is needed when cross-compiling for iOS (Aarch64 + Darwin). I also
changed the syntax of the Linux stdenv for visual consistency, though
that has no effect on semantics as the os is already guaranteed to be
Linux.
This reverts commit 0a944b345e, reversing
changes made to 61733ed6cc.
I dislike these massive stdenv changes with unclear motivation,
especially when they involve gratuitous mass renames like NIX_CC ->
NIX_BINUTILS. The previous such rename (NIX_GCC -> NIX_CC) caused
months of pain, so let's not do that again.
Environment variable filter in substituteAll was not precise and produced
undefined and invalid variable names. Vladimír Čunát tried to fix that in [1],
but `env -0` did not work during Darwin bootstrap, so [2] reverted this change
and replaced an error due to invalid variables with a warning. Recently in #28057
John Ericson added `set -u` to `setup.sh` and undefined variables made the setup
fail during e.g. `nix-build -A gnat` with `setup: line 519: !varName: unbound
variable`.
[1] 62fc8859c1
[2] 81df035429
This becomes necessary if more wrappers besides cc-wrapper start
supporting hardening flags. Also good to make the warning into an
error.
Also ensure interface is being used right: Not as a string, not just in
bash.
- Don't build with libsigsegv by default. The build apparently attempted
to link against it, but it never retained the reference anyway...
- Side effect: stdenv bootstrapping needs no libsigsegv anymore.
- Run checks, but only in the interactive gawk by default on Linux,
so that stdenv bootstrap isn't slowed down (by glibc locales, etc.).
- xz should be no longer needed in inputs, as we have it in stdenvs now.
The whole change was triggered by some used kernel versions still
breaking libsigsegv tests #28464.
Older bash version, like those in the bootstrap tools and on macOS,
currently confuse variables defined as an empty array with undefined
variables. `${foo+"${foo[@]}"}` will prevent `set -u` problems with
empty arrays and older without making a single '' in the empty case.
Care is taken to `set +u` when running hooks so as to not break existing
packages.
This reverts commit eeabf85780.
This change suddenly makes tons of stdenv internals visible in
nativeBuildInputs of every derivation, which doesn't seem desirable.
E.g:
````
nix-repl> hello.nativeBuildInputs
[ «derivation /nix/store/bcfkyf6bhssxd2vzwgzmsbn7b5b9rpxc-patchelf-0.9.drv»
«derivation /nix/store/4wnshnz9wwanpfzcrdd76rri7pyqn9sk-paxctl-0.9.drv»
<< snip 10+ lines >>
«derivation /nix/store/d35pgh1lcg5nm0x28d899pxj30b8c9b2-gcc-wrapper-6.4.0.drv»
]
````
Additionally, instead of pulling them from `setup.sh`, route them via
Nix. This gets us one step closer to making stdenv be a plain attribute
set instead of a derivation.
Currently the logic of generating nixpkgs Hydra jobs is to walk through
the pkgs evaluated for system = "x86_64-linux", collect any derivations
and their meta.platforms values. However, that doesn't work for
packages whose meta.platforms doesn't include x86_64-linux, as just
evaluating their meta attribute raises an error so they get skipped
completely.
As a less-intrusive fix (i.e. anything than rewriting the current package
enumeration logic), allow passing `config.allowUnsupportedSystem = true`
to permit evaluating packages regardless of their platform and use that
in the package listing phase.
Fixes#25200
@vcunat and others rightly point out that it's easier to quote always,
than learn Bash's idiosyncrasies enough to know when it doesn't make a
difference.
This reverts commit 2743078f66, which
removes quotes that don't do anything, and then goes further adding
even more quotes.
The logic was made pure for the normal libSystem, but this change never
made it to the bootstrap tools. Deduplication the logic as the comment
suggests would have prevented this, but here's a stop-gap until we do
so.
It's better than the eval solution this is adding back, but until we can
rely on a particular version of bash in nix-shell, this just breaks too
much stuff.
See c94f3d5575
and https://github.com/NixOS/nix/pull/1483 for the better long-term
solution.
As @oxij points out in [1], this breakage is especially serious because
it changes the contents of built environments without a corresonding
change in their hashes. Also, the revert is easier than I thought.
This reverts commit 3cb745d5a6.
[1]: https://github.com/NixOS/nixpkgs/pull/27427#issuecomment-317293040
`nix-build pkgs/top-level/release.nix -A tarball` now succeeds.
`configureFlags = null` lead to a type error, and one overrideDrv
needed to be converted to to append a configureFlags list instead of
string due to the normalization.
Thanks @vcunat for alerting me to the issues---sorry I did not catch
them before merging my own PR.
Eventually the adapter will be removed. Moved is
- Name suffix from hostPlatform
- configurePlatforms
To not cause more breakage, the default is currently [], but
eventually it will be [ "build" "host" ], as the cross adapter makes
it today.
I took some liberties with the flags-echoing code to make it more
concise and correct. Also, a few warnings in findInputs and friends I
skipped because I am going to rewrite those anyways.
Thanks @grahamc for telling me about this great linter!
The main changes are in libSystem, which lost the coretls component in 10.13
and some hardening changes that quietly crash any program that uses %n in
a non-constant format string, so we've needed to patch a lot of programs that
use gnulib.
This makes those files a bit easier to read. Also, for what it's worth,
it brings us one baby step closer to handling spaces in store paths.
Also, I optimized handling of many transitive deps with read. Probably,
not very beneficial, but nice to enforce the pkg-per-line structure.
Doing so let me find much dubious code and fix it.
Two misc notes:
- `propagated-user-env-packages` also needed to be adjusted as
sometimes it is copied to/from the propagated input files.
- `local fd` should ensure that file descriptors aren't clobbered
during recursion.
Only cosmetic changes are done otherwise.
Real refactoring is left for later.
There's a small slow-down on my machine:
$ time nix-env -qa -P >/dev/null
gets from ~2.8 to ~3.5 seconds (negligible change in RAM).
That's most likely caused by sharing less computation between different
mkDerivation calls, and I plan to improve that soon.
This is a bit simpler now, but more importantly it scales better when I
double the number of sorts of dependencies as part of my cross
compilation work.
Packages get --host and --target by default, but can explicitly request
any subset to be passed as needed. See docs for more info.
rustc: Avoid hash breakage by using the old (ignored)
dontSetConfigureCross when not cross building
This is especially useful when not cross compiling. It means we can
remove the `stdenv.isGlibc` predicate too.
Additionally, use this to simplify the logic to choose the
appropriate libiconv derivation.
`pkgsNoParams` was removed by me, but then #25035 was merged using it,
leading to an unbound identifier.
It would be nice to get travis to do build release-cross.nix or
something to catch these things.
In the extremely unlikely case that our store hash path ends in several
digits (as is the case right now), the Darwin ld will try to interpret
those digits as a version number and barf. To avoid that, we pass in the
SDK version explicitly to stop it from trying to figure it out from iffy
context.
See previous commit for what was done to `binutils` to make this
possible.
There were some uses of `forcedNativePackages` added. The
combination of overrides with that attribute is highly spooky: it's
often important that if an overridden package comes from it, the
replaced arguments for that package come from it. Long term this
package set and all the spookiness should be gone and irrelevant:
"Move along, nothing to see here!"
No hashes should be changed with this commit
Use `buildPackages.binutils` to get build = host != target binutils,
i.e. the old `binutilsCross`, and use
`buildPackages.buildPackages.binutils` to get build = host = target
binutils, i.e. the old `binutils`.
`buildPackages` chains like this are supposed to remove the need for
all such `*Cross` derivations. We start with binutils because it's
comparatively easy.
No hashes of cross-tests should be changed
stdenv.cross is a silly attribute that needs to go leaving the well-defined hostPlatform and targetPlatform. This PR doesn't remove it, but changes its definition: before it tracked the target platform which is sometimes more useful for compilers, and now it tracks the host platform which is more useful for everything else. Most usages are libraries, falling in the "everything else" category, so changing the definition makes sense to appease the majority. The few compiler (gcc in particular) uses that exist I remove to use targetPlatform --- preserving correctness and becoming more explicit in the process.
I would also update the documentation aside mentioning stdenv.cross as deprecated, but the definition given actually erroneously assumes this PR is already merged!
Before all overrides were also pruned in the previous stage, now
only gcc and binutils are, because they alone care about about the
target platform. The rest of the overrides don't, so it's better to
preserve them in order to avoid spurious rebuilds.
Each bootstrapping stage ought to just depend on the previous stage, but
poorly-written compilers break this elegence. This provides an easy-enough
way to depend on the next stage: targetPackages. PLEASE DO NOT USE IT
UNLESS YOU MUST!
I'm hoping someday in a pleasant future I can revert this commit :)
When not cross compiling, nativeBuildInputs and buildInputs have
identical behaviour. Currently that is implemented by having
mkDerivation do a concatenation of those variables in Nix code and pass
that to the builder via the nativeBuildInputs attribute.
However, that has some annoying side effects, like `foo.buildInputs`
evaluating to `[ ]` even if buildInputs were specified in the nix
expression for foo.
Instead, pass buildInputs and nativeBuildInputs in separate variables as
usual, and move the logic of cross compilation vs. native compilation to
the stdenv builder script. This is probably a tiny bit uglier but
fixes the previous problem.
Issue #4855.
If a package's meta has `knownVulnerabilities`, like so:
stdenv.mkDerivation {
name = "foobar-1.2.3";
...
meta.knownVulnerabilities = [
"CVE-0000-00000: remote code execution"
"CVE-0000-00001: local privilege escalation"
];
}
and a user attempts to install the package, they will be greeted with
a warning indicating that maybe they don't want to install it:
error: Package ‘foobar-1.2.3’ in ‘...default.nix:20’ is marked as insecure, refusing to evaluate.
Known issues:
- CVE-0000-00000: remote code execution
- CVE-0000-00001: local privilege escalation
You can install it anyway by whitelisting this package, using the
following methods:
a) for `nixos-rebuild` you can add ‘foobar-1.2.3’ to
`nixpkgs.config.permittedInsecurePackages` in the configuration.nix,
like so:
{
nixpkgs.config.permittedInsecurePackages = [
"foobar-1.2.3"
];
}
b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
‘foobar-1.2.3’ to `permittedInsecurePackages` in
~/.config/nixpkgs/config.nix, like so:
{
permittedInsecurePackages = [
"foobar-1.2.3"
];
}
Adding either of these configurations will permit this specific
version to be installed. A third option also exists:
NIXPKGS_ALLOW_INSECURE=1 nix-build ...
though I specifically avoided having a global file-based toggle to
disable this check. This way, users don't disable it once in order to
get a single package, and then don't realize future packages are
insecure.
If a package's meta has `knownVulnerabilities`, like so:
stdenv.mkDerivation {
name = "foobar-1.2.3";
...
meta.knownVulnerabilities = [
"CVE-0000-00000: remote code execution"
"CVE-0000-00001: local privilege escalation"
];
}
and a user attempts to install the package, they will be greeted with
a warning indicating that maybe they don't want to install it:
error: Package ‘foobar-1.2.3’ in ‘...default.nix:20’ is marked as insecure, refusing to evaluate.
Known issues:
- CVE-0000-00000: remote code execution
- CVE-0000-00001: local privilege escalation
You can install it anyway by whitelisting this package, using the
following methods:
a) for `nixos-rebuild` you can add ‘foobar-1.2.3’ to
`nixpkgs.config.permittedInsecurePackages` in the configuration.nix,
like so:
{
nixpkgs.config.permittedInsecurePackages = [
"foobar-1.2.3"
];
}
b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
‘foobar-1.2.3’ to `permittedInsecurePackages` in
~/.config/nixpkgs/config.nix, like so:
{
permittedInsecurePackages = [
"foobar-1.2.3"
];
}
Adding either of these configurations will permit this specific
version to be installed. A third option also exists:
NIXPKGS_ALLOW_INSECURE=1 nix-build ...
though I specifically avoided having a global file-based toggle to
disable this check. This way, users don't disable it once in order to
get a single package, and then don't realize future packages are
insecure.
This fixes the "sliding window" principle:
0. Run packages: build = native; host = foreign; target = foreign;
1. Build packages: build = native; host = native; target = foreign;
2. Vanilla packages: build = native; host = native; target = native;
3. Vanilla packages: build = native; host = native; target = native;
n+3. ...
Each stage's build dependencies are resolved against the previous stage,
and the "foreigns" are shifted accordingly. Vanilla packages alone are
built against themsevles, since there are no more "foreign"s to shift away.
Before, build packages' build dependencies were resolved against
themselves:
0. Run packages: build = native; host = foreign; target = foreign;
1. Build packages: build = native; host = native; target = foreign;
2. Build packages: build = native; host = native; target = foreign;
n+2. ...
This is wrong because that principle is violated by the target
platform staying foreign.
This will change the hashes of many build packages and run packages, but
that is OK. This is an unavoidable cost of fixing cross compiling.
The cross compilation docs have been updated to reflect this fix.
Our bootstrap tools are actually broken right now due to busybox not
working when invoked directly from a store path. (It says e.g.
"0qqqw19y4gmknajw8vg4fvhx9gxdqlhz-busybox: applet not found").
Make this test actually fail in such case, the next commit will fix the
problem with busybox.
This is required for Aarch64 since a lot of source tarballs ship with
outdated configure scripts that don't recognize aarch64. Simply
replacing the config.guess and config.sub with new versions from
upstream makes them build again.
This same approach is used by at least Buildroot and Fedora. In
principle this could be enabled for all architectures but
conditionalizing this on aarch64 avoids a mass rebuild on x86.
The long term goal is a big replace:
{ inherit system platform; } => buildPlatform
crossSystem => hostPlatform
stdenv.cross => targetPlatform
And additionally making sure each is defined even when not cross compiling.
This commit refactors the bootstrapping code along that vision, but leaves
the old identifiers with their null semantics in place so packages can be
modernized incrementally.
[N.B., this package also applies to the commits that follow it in the same
PR.]
In most cases, buildPackages = pkgs so things work just as before. For
cross compiling, however, buildPackages is resolved as the previous
bootstrapping stage. This allows us to avoid the mkDerivation hacks cross
compiling currently uses today.
To avoid a massive refactor, callPackage will splice together both package
sets. Again to avoid churn, it uses the old `nativeDrv` vs `crossDrv` to do
so. So now, whether cross compiling or not, packages with get a `nativeDrv`
and `crossDrv`---in the non-cross-compiling case they are simply the same
derivation. This is good because it reduces the divergence between the
cross and non-cross dataflow. See `pkgs/top-level/splice.nix` for a comment
along the lines of the preceding paragraph, and the code that does this
splicing.
Also, `forceNativeDrv` is replaced with `forceNativePackages`. The latter
resolves `pkgs` unless the host platform is different from the build
platform, in which case it resolves to `buildPackages`. Note that the
target platform is not important here---it will not prevent
`forcedNativePackages` from resolving to `pkgs`.
--------
Temporarily, we make preserve some dubious decisions in the name of preserving
hashes:
Most importantly, we don't distinguish between "host" and "target" in the
autoconf sense. This leads to the proliferation of *Cross derivations
currently used. What we ought to is resolve native deps of the cross "build
packages" (build = host != target) package set against the "vanilla
packages" (build = host = target) package set. Instead, "build packages"
uses itself, with (informally) target != build in all cases.
This is wrong because it violates the "sliding window" principle of
bootstrapping stages that shifting the platform triple of one stage to the
left coincides with the next stage's platform triple. Only because we don't
explicitly distinguish between "host" and "target" does it appear that the
"sliding window" principle is preserved--indeed it is over the reductionary
"platform double" of just "build" and "host/target".
Additionally, we build libc, libgcc, etc in the same stage as the compilers
themselves, which is wrong because they are used at runtime, not build
time. Fixing this is somewhat subtle, and the solution and problem will be
better explained in the commit that does fix it.
Commits after this will solve both these issues, at the expense of breaking
cross hashes. Native hashes won't be broken, thankfully.
--------
Did the temporary ugliness pan out? Of the packages that currently build in
`release-cross.nix`, the only ones that have their hash changed are
`*.gcc.crossDrv` and `bootstrapTools.*.coreutilsMinimal`. In both cases I
think it doesn't matter.
1. GCC when doing a `build = host = target = foreign` build (maximally
cross), still defines environment variables like `CPATH`[1] with
packages. This seems assuredly wrong because whether gcc dynamically
links those, or the programs built by gcc dynamically link those---I
have no idea which case is reality---they should be foreign. Therefore,
in all likelihood, I just made the gcc less broken.
2. Coreutils (ab)used the old cross-compiling infrastructure to depend on
a native version of itself. When coreutils was overwritten to be built
with fewer features, the native version it used would also be
overwritten because the binding was tight. Now it uses the much looser
`BuildPackages.coreutils` which is just fine as a richer build dep
doesn't cause any problems and avoids a rebuild.
So, in conclusion I'd say the conservatism payed off. Onward to actually
raking the muck in the next PR!
[1]: https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
This patch add a new argument to Nixpkgs default expression named "overlays".
By default, the value of the argument is either taken from the environment variable `NIXPKGS_OVERLAYS`,
or from the directory `~/.nixpkgs/overlays/`. If the environment variable does not name a valid directory
then this mechanism would fallback on the home directory. If the home directory does not exists it will
fallback on an empty list of overlays.
The overlays directory should contain the list of extra Nixpkgs stages which would be used to extend the
content of Nixpkgs, with additional set of packages. The overlays, i-e directory, files, symbolic links
are used in alphabetical order.
The simplest overlay which extends Nixpkgs with nothing looks like:
```nix
self: super: {
}
```
More refined overlays can use `super` as the basis for building new packages, and `self` as a way to query
the final result of the fix-point.
An example of overlay which extends Nixpkgs with a small set of packages can be found at:
https://github.com/nbp/nixpkgs-mozilla/blob/nixpkgs-overlay/moz-overlay.nix
To use this file, checkout the repository and add a symbolic link to
the `moz-overlay.nix` file in `~/.nixpkgs/overlays` directory.
`gcc-unwrapped` basically replaces `gccPlain`. It may seem like an ugly
polution to stick it in all-packages, but a future PR will enshrine this
`*-unwrapped` pattern. In any event, the long term goal is stdenvs might
need to tweak how compilers are booted and wrapped, but the code to build
the unwrapped compilers themselves should be generic.
Introduce new abstraction, `stdenv/booter.nix` for composing bootstraping
stages, and use it everywhere for consistency. See that file for more doc.
Stdenvs besides Linux and Darwin are completely refactored to utilize this.
Those two, due to their size and complexity, are minimally edited for
easier reviewing.
No hashes should be changed.
On one hand, don't want to pass garbage that affects hash, on the other
hand footguns are bad.
Now, factored out the derivation so only need to pass in what is used.
- The darwin test can now force the use of the freshly-booted darwin stdenv
- The linux test now passes enough dummy arguments
This may make debugging harder, if so, check out #20889
- Non-cross stdenvs are honest and assert that `crossSystem` is null
- `crossSystem` is a mandatory argument to top-level/stage.nix, just like
`system` and `platform`
- Broken default arguments on stdenvs for testing are gone.
- All stdenvs (but little-used stdenvNix) take the same arguments for easy
testing.
This makes the flow of data easier to understand. There's little downside
because the args in question are already inspected by the stdenvs.
cross-compiling in particular is simpler because we don't need to worry
about overriding the config closed over by `allPackages`.
This commit changes the dependencies of stdenv, and clean-up the stdenv
story by removing the `defaultStdenv` attribute as well as the `bootStdenv`
parameter.
Before, the final bootstrapping stage's stdenv was provided by
all-packages, which was iterating multiple times over the
top-level/default.nix expression, and non-final bootstrapping stages'
stdenvs were explicitly specified with the `bootStdenv` parameter.
Now, all stages' stdenvs are specified with the `stdenv` parameter.
For non-final bootstrapping stages, this is a small change---basically just
rename the parameter.
For the final stage, top-level/default.nix takes the chosen stdenv and
makes the final stage with it.
`allPackages` is used to make all bootstrapping stages, final and
non-final alike. It's basically the expression of `stage.nix` (along with a
few partially-applied default arguments)
Note, the make-bootstrap-tools scripts are temporarily broken
This reinstates the libSystem selective symbol export machinery we used
to have, but locks it to the symbols that were present in 10.11 and skips
the actual compiled code we put into that library in favor of the system
initialization code. That should make it more stable and less likely to
do weird stuff than the last time we did this.
`stripHash` documentation states that it prints out the stripped name to
the stdout, but the function stored the value in `strippedName`
instead.
Basically all usages did something like
`$(stripHash $foo | echo $strippedName)` which is just braindamaged.
Fixed the implementation and all invocations.
This is similar to `overrideDerivation`, but overrides the arguments to
`mkDerivation` instead of the underlying `derivation` call.
Also update `makeOverridable` so that uses of `overrideAttrs` can be
followed by `override` and `overrideDerivation`, i.e. they can be
mix-and-matched.
The 3.8 upgrade was premature and appears to be breaking a load of stuff
that I can't look into right now. Will take it back to 3.8 and test more
thoroughly before pushing more broadly. I should learn to change fewer
variables at a time :)
It's a long build and generally painful to split into smaller commits,
so I apologize for lumping many changes into one commit but this is far
easier.
There are still several outdated parts of the darwin stdenv but these
changes should bring us closer to the goal.
Fixes#18461
I broke this in the cleanups I did in 171c7f0, the gcc inside the bootstrap
tarball is not getting built with the correct --with-fpu, --with-float
etc. options.
Revert a revert of a merge that shouldn't have been in master but was intentionally in staging.
Next time I'll do this right after the revert instead of so far down the line...
This reverts commit 9adad8612b.
This will break part of the bootstrap tools tests because the new tools
need some changes in the stdenv, but if I change them all at once, the
stdenv breaks with the old bootstrap tools. So I'm doing this first, then
will make changes to the stdenv once this bundle is built and I can use
it.
I also added some functionality to let me test one set of bootstrap tools
on another nixpkgs tree, which makes testing a lot more pleasant.
- cloog, ppl, cloogppl aren't used by recent GCCs. Kill references to them.
- Use correct versions of isl, as the current GCC depends
on non-default versions of them.
- Also clarify isl dynamic libraries are needed in cross
builds, but not in native builds
- Since aeb3d8c (bzip2: fix cross build on mingw by using autoconf patch),
it seems that the bzip2 binary depends on libbz2 when cross compiling.
So copy libbz2 into the bootstrap tarball as well.
- Curl isn't used in the bootstrap tools since e6f61b4cf3.
This reverts commit debd401b0f.
We must not use a single-binary build for the bootstrap since the common
binary gains a dynamic linkage to gmp (due to 'factor' and 'expr'
handling arbitrary-precision arithmetic).
Our coreutils now uses single-binary-build mode where, by default,
simple shebang scripts are used for all the binaries. That doesn't work
e.g. with the Linux unpacker which only handles standard binaries and
symlinks. Let's use the symlinked mode instead for boostrapping.
This does NOT change any stdenv hashes.
I only tested the case most important to me:
$ nix-build pkgs/top-level/release.nix -A stdenvBootstrapTools.x86_64-linux.test
Close#15803. This avoids the error:
while setting up the build environment: executing
‘/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin/bash’:
Argument list too long
Note: I wanted to make it optional based on buildCommand length,
but that seems pointless as I'm sure it's less performant.
Amended by vcunat:
https://github.com/NixOS/nixpkgs/pull/15803#issuecomment-224841225
On Linux, paxctl's setup hook should overwrite the paxmark stub, but the
stub is defined after the setup hooks are sourced, so the stub ends up
overwriting the real function. The result is that paxmark fails to do
anything. The fix is to define the stub before any setup hooks are
sourced. Thanks to @vcunat for figuring this out.
Closes#15492
This changes cygwin stdenv, but I don't think it will hurt much people.
This allows mkDerivation to get "dontRebase=true" to skip the usual cygwin
rebase. This is required, if we are using this stdenv to build DLLs for win32
inside x86_64-cygwin, because /bin/rebase crashes at finding an arch mismatch.
Additionally, we don't need any rebase for libraries built by visual studio and
meant for visual studio (my use case).
I'm using nix in x86_64-cygwin to build libraries with visual studio, both for
x86_64 and x86.
I'm giving this up. Feel free to find some reasonable variant that works
at least on Linux and Darwin. Problems encountered:
- During bootstrap of Darwin stdenv `env -0` and some bash features
don't work.
- Without `env -0` the contents of some multi-line phases is taken as
variable declarations, which wouldn't typically matter, but the PR
wanted to refuse bash-invalid names which would be occasionally
triggered. This commit dowgrades that to a warning with explanation.
It turned out that process substitution fed into a while-cycle
isn't recognized during darwin bootstrap:
http://hydra.nixos.org/build/35382446/nixlog/1/raw
Also fix broken NIX_DEBUG output, noticed by abbradar.
The set/env fix in #14907 wasn't very good, so let's use a null-delimited
approach. Suggested by Aszlig.
In particular, this should fix a mass-breakage on Darwin, though I was
unable to test that.
bash variable names may only contain alphanumeric ASCII-symbols and _,
and must not start with a number. Nix expression attribute names however
might contain nearly every character (in particular spaces and dashes).
Previously, a substitution that was not a valid bash name would be
expanded to an empty string. This commit introduce a check that throws
a (hopefully) helpful error when a wrong name is used in a substitution.
For every *.{exe,dll} in $output/bin/ we try to find all (potential)
transitive dependencies and symlink those DLLs into $output/bin
so they are found on invocation.
(DLLs are first searched in the directory of the running exe file.)
The links are relative, so relocating whole /nix/store won't break them.
The hook is activated on cygwin and when cross-compiling to mingw.
