Turns out markers are non-deterministic after all and even our patching still
doesn't solve this problem completely. For example (tensorflow deps, this is a
complete diff so actual dependencies don't differ):
30509c30509
< bc527ff00916b15caee38793bca8f294c748df4a256de55c5199281be0489e73 result/@bazel_skylib.marker
---
> 4e0303e815c78df1e43d4b88dfe65e73046e0c6157fb10aa9a4e8b910113cd9c result/@bazel_skylib.marker
31045c31045
< fa13d04b2316214c3b4008b52546c2d5b633e006f6f019d597bb3f9745bacf7b result/@bazel_toolchains.marker
---
> b36174bf5535e5157801b6de30c35ee03a03fe57766306393c3d65dd65cbebf4 result/@bazel_toolchains.marker
31144c31144
< b0ce4a3ac29ac22528336dd3a54b5b7af9ecc43bef2a2630713c1981a5cbbb51 result/@build_bazel_rules_swift.marker
---
> 7492528068ec4f8e7ace2ecf8f933ec4e1b2235bd7426ce6f70177919f1cd05e result/@build_bazel_rules_swift.marker
36245c36245
< be2993536a8233d63251b664caf35b1e7cd57d194ab2a39a293876c232d6bbd0 result/@io_bazel_rules_closure.marker
---
> b6655cc3f2c78525e5a724d8a4e93b1e7f09f1e09fc817d231109e7f39103e88 result/@io_bazel_rules_closure.marker
36329c36329
< 087bc674c9509dfe157400d111db4a13eeb45fc76aeccd490cee9aad6771ecad result/@io_bazel_rules_docker.marker
---
> f920ec07315ec71e800b05cd22b2a341c0a80807c6e335ee81739b13c532b422 result/@io_bazel_rules_docker.marker
79544d79543
< 85893a05a817036c61f6cd9f8247757baa1654f473c494ce4fc5253c2bbd2790 result/@platforms.marker
And here's an example of differences:
$ cat result-a/@bazel_skylib.marker
7dc7472d37424ba5ec6a5532765bc911
$MANAGED
cat result-b/@bazel_skylib.marker
a8f3f577798201157128e8e9934c4705
$MANAGED
Instead of trying to patch these markers further we now completely clear them.
Nix hacks for ignoring markers are restored and expanded so that we don't even
attempt to parse the marker.
* buildBazelPackage: autodetect nix toolchain instead of Xcode on Darwin
* do not export the variables outside of Darwin
* remove unecessary parens
* move comment within the darwin check
Timestamp verification skip is no longer needed (not sure why). Generally we
better off always using the environment hack for all packages because that
ensures all NIX_* flags are correctly applied.
One possible improvement in future is to filter only NIX_* variables to
passthru in Bazel.
Before one would get the following error
nix-repl> pkgs.writeTextDir "share/my-file" "foo"
error: invalid character '/' in name 'share/my-file'
Fixes#50347
Previously the installPhase of the fixed ouput derivation would fail for
a package that has no markers, since `sed` would complain about having
no input files. If we use `find` instead of bash globs, that problem
goes away.
This avoids dumping -Wall warnings when they appear in framework
headers. As a result, we are closer to how regular headers are
included (via -isystem).
Also remove ccIncludeFlag lookup, this was unused & not very useful.
mergeInputs is now simply defined in terms of `concatLists` and
`catAttrs` instead of a more complicated `foldr`.
Note that the order of PATH has also changed. For example running the
following with nix-shell:
let
pkgs = import <nixpkgs> {};
shell1 = pkgs.mkShell {
buildInputs = [ pkgs.htop ];
};
shell2 = pkgs.mkShell {
buildInputs = [ pkgs.hello ];
};
shell3 = pkgs.mkShell {
inputsFrom = [ shell1 shell2 ];
buildInputs = [ pkgs.tree ];
};
in shell3
Results in the following PATH:
$ echo $PATH
...
/nix/store/yifq4bikf7m07160bpia7z48ciqddbfi-tree-1.8.0/bin:
/nix/store/vhxqk81234ivqw1a7j200a1c69k8mywi-htop-2.2.0/bin:
/nix/store/n9vm3m58y1n3rg3mlll17wanc9hln58k-hello-2.10/bin
...
Previously the order was:
/nix/store/n9vm3m58y1n3rg3mlll17wanc9hln58k-hello-2.10/bin
/nix/store/vhxqk81234ivqw1a7j200a1c69k8mywi-htop-2.2.0/bin:
/nix/store/yifq4bikf7m07160bpia7z48ciqddbfi-tree-1.8.0/bin:
I think the new order makes more sense because it allows to override
the PATH in the outermost mkShell.
Running the following expression with nix-shell:
let
pkgs = import <nixpkgs> {};
shell1 = pkgs.mkShell {
shellHook = ''
echo shell1
'';
};
shell2 = pkgs.mkShell {
shellHook = ''
echo shell2
'';
};
shell3 = pkgs.mkShell {
inputsFrom = [ shell1 shell2 ];
shellHook = ''
echo shell3
'';
};
in shell3
Will now results in:
shell2
shell1
shell3
Note that packages in the front of inputsFrom have precedence over
packages in the back. The outermost mkShell has precedence over all.
The previous behaviour would work fine as long as `symlink` is a link to
a file. If is a link to a directory though, the new `ln` wouldn't
overwrite it but would create a new link *in that directory* (with the
name of the link source).
Instead, we can precompute the target location, then first remove the
symlink and write the new one in its place.
After bumping sublime3 in #61636 we realized that saving files as root
doesn’t work anymore and somehow the paths weren’t patched by
`libredirect`.
After some debugging it came out that Sublime switched from `posix_spawn(3)`
to `posix_spawnp(3)` to start new processes internally. Since `libredirect`
only handled the former, `/usr/bin/pkexec` stopped being redirected.
Wrapping `posix_spawnp` fixes the problem.
PR #58431 added /nix/store to each layer.tar. However, the timestamp was
not explicitly set while adding /nix and /nix/store to the archive. This
resulted in different SHA256 hashes of layer.tar between image builds.
This change sets time and owner when tar'ing /nix/store.
To avoid symlink loops to /host in nested chrootenvs we need to remove
one level of indirection. This is also what's generally expected of
/host contents.
* Remove unused argument from pivot_root;
* Factor out tmpdir creation into a separate function;
* Remove unused fstype from bind mount;
* Use unlink instead of a treewalk to remove empty temporary directory.
The problem with stacking chrootenv before was that CLONE_NEWUSER cannot
be used when a child uses chroot. So instead of that we use pivot_root
which replaces root in the whole namespace. This requires our new root
to be an actual fs so we mount tmpfs.
not all valid file names are valid derivation names. This can cause
troubles when, for example, trying to place systemd template unit
files, which contain an '@' in their name, in an initrd.
Fixes#53987
glibc 2.27 (and possibly other versions) can't handle an `nopenfd` value larger than 2^19 in `ntfw`, which is problematic if you've set the maximum number of fds per process to a value higher than that.
Commit "patchShebangs: Allow for multiple arguments" 4a1e51f957
removed the check. We don't want to break existing usages so this
introduces it again with a successful exit code.
It's tempting to think patchShebangs supports multiple arguments.
Without this patch it just silently ignores all but the first. Now it
patches the shebangs in all of its arguments.
Fixes: #57695
This hook allows to add NixOS driver libraries path to given ELF
objects' RUNPATH. We use it instead of settings RUNPATH manually
everywhere. It must be invoked in postFixup so that RUNPATH stripping
does not remove the path.
It puts the path first instead of last so that system-wide drivers
are always preferred.
The layer order was not correct when a parent image was used: parent
image layers were above the new created layer.
This commits simplifies the code related to layer ordering. In
particular, layers in `layer-list` are ordered from bottom-most to
top-most. This is also the order of layers in the `rootfs.diff_ids`
attribute of the image configuration.
While it might be useful in some cases, there are too many caveats to be worth it.
When libredirect intercepts dlopen call and calls the original function, the dynamic
loader will use libredirect.so's DT_RUNPATH entry instead of the one from the ELF file
the dlopen call originated from. That means that when program tries to dlopen a library
that it expects to find on its RPATH, the call will fail.
This broke Sublime Text for just that reason.
Especially as a new user it is a much better experience to receive a
proper help response to `-h`. Currently passing `-h` will cause some
runtime error with the `git remote` error help being shown. Not very
helpful.
It doesn't hurt to be a bit more user friendly in this case.
This introduces extra escaping for $NIX_STORE that gets interpolated
into a PCRE. The escaping is performed using a standard Perl function
"quotemeta" (see "perldoc -f quotemeta" for reference). The same value
is also used in sed regex which uses POSIX basic regular expressions
instead of PCRE, so it needs fewer characters to be escaped. It should
not cause much problem to not change sed invocation, but I replace it
with equivalent Perl expression (actually the behavior is changed to not
output a newline character after the matched output).
We want to make sure this value is explicitly set. Infering it for
every arch leads to annoying failures like:
https://hydra.nixos.org/build/92583832/
Perhaps we can enable it in the future with some smarter handling of
cc-wrapper.sh.
Creating the timestamp in the patched script's directory has a few
drawbacks:
* if "foo.timestamp" already exists, it will be overwritten
* it requires the directory to be writable
Adds pkgsCross.wasm32 and pkgsCross.wasm64. Use it to build Nixpkgs
with a WebAssembly toolchain.
stdenv/cross: use static overlay on isWasm
isWasm doesn’t make sense dynamically linked.
It is useful to make these dynamic and not bake them into gcc. This
means we don’t have to rebuild gcc to change these values. Instead, we
will pass cflags to gcc based on platform values. This was already
done hackily for android gcc (which is multi-target), but not for our
own gccs which are single target.
To accomplish this, we need to add a few things:
- add ‘arch’ to cpu
- add NIX_CFLAGS_COMPILE_BEFORE flag (goes before args)
- set -march everywhere
- set mcpu, mfpu, mmode, and mtune based on targetPlatform.gcc flags
cc-wrapper: only set -march when it is in the cpu type
Some architectures don’t have a good mapping of -march. For instance
POWER architecture doesn’t support the -march flag at all!
https://gcc.gnu.org/onlinedocs/gcc/RS_002f6000-and-PowerPC-Options.html#RS_002f6000-and-PowerPC-Options
gobject-introspection uses glib’s g_module_open function, which in turn relies
on dlopen. I also implemented openat, since I initially thought this function
was used but turns out dlopen uses the openat signal directly. We might as
well keep it, even thought I do not need it at the moment.
this adds libc++ to the LLVM cross, giving us access to the full
Nixpkgs set. This requires 4 stages of wrapped compilers:
- Clang with no libraries
- Clang with just compiler-rt
- Clang with Libc, and compiler-rt
- Clang with Libc++, Libc, and compiler-rt
This iteration was long, about five weeks (2fcb11a2), I think.
Darwin: it's missing a few thousand binaries and there's a make-netbsd
regression, but I suppose these aren't merge blockers.
buildRustCrate has a handy `include` helper, that only imports those whitelisted
files and folders to the store.
However, the function's matching logic is broken and includes all files,
regardless of whether or not they're whitelisted, as long as the whitelist
contains at least one name (regardless of whether that name exists). This is
because it doesn't take into account that
`lib.strings.removePrefix "foo" "bar" == "bar"` (that is, paths that don't match
the prefix are passed straight through).
This rare sitation was caught when building zoom-us package:
```
automatically fixing dependencies for ELF files
/nix/store/71d65fplq44y9yn2fvkpn2d3hrszracd-auto-patchelf-hook/nix-support/setup-hook: line 213: echo: write error: Broken pipe
/nix/store/71d65fplq44y9yn2fvkpn2d3hrszracd-auto-patchelf-hook/nix-support/setup-hook: line 210: echo: write error: Broken pipe
```
The worst is that derivation continued and resulted into broken package:
https://github.com/NixOS/nixpkgs/pull/55566#issuecomment-470065690
I hope, replacing `grep -q` with `grep` will remove this race condition.
When a package provides both executables and gio modules, it is quite
probable the executables will need those modules. wrapGAppsHook wraps
executables with GIO_EXTRA_MODULES picked up from dependencies
but forgets about the package being built. Let’s add to consideration.
Closes: https://github.com/NixOS/nixpkgs/issues/50254
On very large graphs (14k+ paths), we'd end up with a massive in
memory tree of mostly duplication.
We can safely cache trees and point back to them later, saving
memory.
While it is not obvious from the source, cargo sets CARGO_MANIFEST_DIR to an absolute directory. This let to a build problem with the popular "tera" crate using the "pest" crate.
## Cargo details
The variable is set here:
f7c91ba622/src/cargo/core/compiler/compilation.rs (L229)
and computed from the `manifest_path`:
f7c91ba622/src/cargo/core/package.rs (L163)
The manifest path is also exported via `cargo metadata` where you can see that it is absolute.
Whenever we create scripts that are installed to $out, we must use runtimeShell
in order to get the shell that can be executed on the machine we create the
package for. This is relevant for cross-compiling. The only use case for
stdenv.shell are scripts that are executed as part of the build system.
Usages in checkPhase are borderline however to decrease the likelyhood
of people copying the wrong examples, I decided to use runtimeShell as well.
