We were using 'Combined Image JSON + Filesystem Changeset Format' [1] to
unpack and pack image and this patch switches to the format used by the registry.
We used the 'repository' file which is not generated by Skopeo when it
pulls an image. Moreover, all information of this file are also in the
manifest.json file.
We then use the manifest.json file instead of 'repository' file. Note
also the manifest.json file is required to push an image with Skopeo.
Fix#29636
[1] 749d90e10f/image/spec/v1.1.md (combined-image-json--filesystem-changeset-format)
The database dump doesn't contain sha and size. This leads to invalid
path in the container. We have to fix the database by using
nix-store.
Note a better way to do this is available in Nix 1.12 (since the
database dump contains all required information).
We also add content output paths in the gcroots since they ca be used
by the container.
Currently, the contents closure is copied to the layer but there is no
nix database initialization. If pkgs.nix is added in the contents,
nix-store doesn't work because there is no nix database.
From the contents of the layer, this commit generates and loads the
database in the nix store of the container. This only works if there
is no parent layer that already have a nix store (to support several
nix layers, we would have to merge nix databases of parent layers).
We also add an example to play with the nix store inside the
container. Note it seems `more` is a missing dependency of the nix
package!
1. `crossDrv` is now the default so we don't need to worry about that in
build != host builds.
2. shell is the build time shell, so `wrapCCCross` doesn't need to
worry, as build == host.
3. `shell.shellPath` will always be appended where useful.
4. Complicated `shell == ""` logic served no purpose.
Fetch into $out and remove all version control files to make it
deterministic (.repo and all .git subdirectories - e.g. the .git/index
files change every time).
Additionally I've changed the default of "useArchive" to false because
fetching with "--archive" will fail for some projects (e.g.
"platform/external/iosched" from the AOSP).
Now, this function should hopefully work for every tag of the AOSP.
Before this patch, a VM was used to spawn docker that pulled the
VM. Now, the tool Skopeo does this job well so we can simplify our
dockerTools since we doesn't need Docker anymore:)
This also fixe the regression described in
https://github.com/NixOS/nixpkgs/issues/29271 : cntlm proxy doesn't
work in 17.09 while it worked in 17.03.
Note Skopeo doesn't produce the same output than docker pull so, we
have to update sha.
This was a problem when run inside a sandbox, e.g. via
"fetchRepoProject". The error message from repo seems unrelated:
fatal: Cannot get https://gerrit.googlesource.com/git-repo/clone.bundle
fatal: error no host given
But the exception is actually thrown due to missing certificates
(/etc/ssl/certs). It should be possible to provide another location via
environment variables (e.g. SSL_CERT_FILE, REQUESTS_CA_BUNDLE or
CURL_CA_BUNDLE) but apparently that doesn't actually work for some
reason (would have to study our Python packaging).
Now "fetchRepoProject" works without the "--no-clone-bundle" option.
The verification was failing with the following error:
gpg: keyblock resource '/tmp/nix-build-XYZ.drv-0/.repo/repo/./.repoconfig/gnupg/pubring.kbx': No such file or directory
Using an absolute path for $HOME fixes this.
And since 175ecbab91 the dependencies on
"git" and "gnupg" aren't required anymore as "gitRepo" already covers
them.
This reverts commit 0a944b345e, reversing
changes made to 61733ed6cc.
I dislike these massive stdenv changes with unclear motivation,
especially when they involve gratuitous mass renames like NIX_CC ->
NIX_BINUTILS. The previous such rename (NIX_GCC -> NIX_CC) caused
months of pain, so let's not do that again.
cctool's as needs to be told use to use gnu as, or else we'd need a
dependency cycle between cctools and clang for this case.
In general, this is not a problem because clang uses its own integrated
assembler where possible, and gnu as otherwise.
To wait for the docker deamon, curl requests are sent. However, if a
http proxy is set, it will respond instead of the docker daemon.
To avoid this, we send docker ps command instead of curl command.
This becomes necessary if more wrappers besides cc-wrapper start
supporting hardening flags. Also good to make the warning into an
error.
Also ensure interface is being used right: Not as a string, not just in
bash.
ccPath is only defined below, so this condition would never be true.
Worse, that's not quite true: what if somebody happend to have `/clang`
and no sandboxing. Boy, wouldn't that be annoying to debug!
set-source-date-epoch-to-latest.sh to ignore files newer than "$NIX_BUILD_TOP/.." (unlike "$NIX_BUILD_TOP" it is root-owned and cannot be touched by nixbld1).
Having multiple compilers in the build environment would result in an
invalid LD_DYLD_PATH like /usr/lib/dyld/usr/lib/dyld.
Since the path is hardcoded in XNU it can't be anything but
/usr/lib/dyld anyway.
This fixes a bug introduced in #27831: `for path in "$dir"/lib*.so` assumed that
all libs match `lib*.so`, but 07674788d6 started
adding libs that match `*.so` and `*.so.*`.
`makeWrapper` and `wrapProgram` are being invoked on all kinds of
wacky things (usually with the help of bash globs or other machine
assistance).
So far, I have come across `wrapProgram` being invoked on a directory,
as well as on the empty string.
As far as I can tell, it's only valid to invoke these utilities on a
normal (non-directory, non-device) executable file. This commit
enforces that precondition.
Previously, makeWrapper would accept arguments it didn't recognize,
potentially allowing argument misspellings or broken callers.
Now, makeWrapper dies with a backtrace if it is called incorrectly.
Also changes `wrapProgram` so that it doesn't pass through the first
argument twice --- this was tripping up the argument checking.
Now is an opportune time to do this, as the infixSalt conversion in
`add-flags.sh` ensures that all the relevant `NIX_*` vars will be
defined even if empty.
This is basically a sed job, in preparation of the next commit. The
rules are more or less:
- s"NIX_(.._WRAPPER_)?([a-zA-Z0-9@]*)"NIX_\1@infixSalt@_\2"g
- except for non-cc-wrapper-specific vars like `NIX_DEBUG`
This is an ugly temp hack for cross compilation, but now we have something better on the way.
Bind `infixSalt` as an environment variable as it will be used in it.
Some programs store the executable in a different place and link it
from the `bin` directory. For example, Polari links `$out/bin/polari`
to `$out/share/polari/org.gnome.Polari`. `wrapGAppsHook` did not follow
symlinks so it was not able to wrap Polari, making it unable to access
GObject introspection definitions required for running the program.
I made the wrapping script follow symlinks to fix this corner case.
In 8d76eff, @Ericson2314 changed the representation of the value that
`findInputs` generated from a whitespace-separated bunch strings to an
actual array of strings.
Expressions that *consume* that value, however, also needed to be
changed to iterate over all the contents of the array, else they would
only select the first value, which turns out to be somewhat limiting.
Fixes#27873
The image json is not exactly the same as the layer json, therefore I
changed the implementation to use the `baseJson` which doesn’t include
layer specific details like `id`, `size` or the checksum of the layer.
Also the `history` entry was missing in the image json. I’m not totally
sure if this field is required, but a I got an error from a docker
registry when I’ve tried to receive the distribution manifest of an
image without those `history` entry:
GET: `http://<registry-host>/v2/<imageName>/manifests/<imageTag>`
```json
{
"errors": [
{
"code": "MANIFEST_INVALID",
"message": "manifest invalid",
"detail": {}
}
]
}
```
I’ve also used a while loop to iterate over all layers which should make
sure that the order of the layers is correct. Previously `find` was
used and I’m not sure if the order was always correct.
callPackage already calls makeOverridable, but that just
makes the function that evaluates to buildEnv overridable,
not buildEnv itself.
If no overridable version of buildEnv is used during construction,
users can't override e.g. `paths` at all
Unified processing of command line arguments in ld-wrapper broke support for
`NIX_DONT_SET_RPATH` and revealed that ld-wrapper adds the directory of its
`-plugin` argument to runpath. This pull request fixes that. It treats
`dir/libname.so` as `-L dir -l name`, because this is how `ld.so` interprets
resulting binary: with `dir` in `RUNPATH` and the bare `libname.so` (without
`dir`) in `NEEDED`, it looks for `libname.so` in each `RUNPATH` and chooses the
first, even when the linker was invoked with an absolute path to `.so`.
As described in https://github.com/NixOS/nixpkgs/issues/18461, MacOS no
longer accepts dylibs which only reexport other dylibs, because their
symbol tables are empty. To get around this, we define an object file
with a single "private extern" symbol, which hopefully won't clobber
anything.
If the base image has been built with nixpkgs.dockerTools, the image
configuration and manifest are readonly so we first need to change
their permissions before removing them.
Fix#27632.
The time to expand rpath was proportional to the number of -L flags times the
number of -l flags. Now it is proportional to their sum (assuming constant
number of files in each directory in an -L flag).
Issue reported by @nh2 at https://github.com/NixOS/nixpkgs/issues/27609#issuecomment-317916623
This setup hook is propagated by gdb. Thus, a typical use is:
$ nix-shell -p gdb nix nix.debug sqlite.debug ...
and gdb will be able find the debug symbols of nix etc. automatically.
The docker loading (docker 1.12.6) of an image with uppercase in the
name fails with the following message:
invalid reference format: repository name must be lowercase
As @oxij points out in [1], this breakage is especially serious because
it changes the contents of built environments without a corresonding
change in their hashes. Also, the revert is easier than I thought.
This reverts commit 3cb745d5a6.
[1]: https://github.com/NixOS/nixpkgs/pull/27427#issuecomment-317293040
Fixes#27406.
Commit 5d4efb2c81 added an assertion to `stopNest'
which requires it be correctly paired with `startNest'. `fetchurl' calls
`stopNest', but never calls `startNest'; the former calls are removed.
The `DISPLAY` environment variable is propagated into chroots built with
`buildFHSUserEnv`, but currently the `XAUTHORITY` variable is not. When
the latter is set, its value is usually necessary in order to connect to
the X server identified by the former.
This matters for users running gdm3, for example, who have `XAUTHORITY`
set to something like `/run/user/1000/gdm/Xauthority` instead of the X
default of `~/.Xauthority`, which doesn't exist in that setup.
Fixes#21532.
Since 3cb745d5a6, the format of
propagated-user-env-packages has changed and propagated packages have not been
included by buildenv, including in the system environment.
The buildenv builder is modified to read propagated-user-env-packages
line-by-line, instead of expecting all packages on one line.
Besides deduplicating overlapping logic, clear warning messages were
added for:
- No glob/path for dynamic linker provided (use default glob)
- Glob did not expand to anything (don't append flag)
- glob expanded to multiple things (take first, like before)
This makes those files a bit easier to read. Also, for what it's worth,
it brings us one baby step closer to handling spaces in store paths.
Also, I optimized handling of many transitive deps with read. Probably,
not very beneficial, but nice to enforce the pkg-per-line structure.
Doing so let me find much dubious code and fix it.
Two misc notes:
- `propagated-user-env-packages` also needed to be adjusted as
sometimes it is copied to/from the propagated input files.
- `local fd` should ensure that file descriptors aren't clobbered
during recursion.
They're additional commits from #26877.
Changing names of the fetched stuff was changing very many hashes,
and I think it's better to avoid that for the moment to reduce work
needed by nixpkgs users. The fetchers are expected to be commonly
used even outside nixpkgs, and the current naming wasn't that bad
usually.
(commit analogical to d10c3cc5eedf58e80e2; I haven't noticed the part of
the PR has already got to master)
The freedesktop specification says that the autostart folder is located at $XDG_CONFIG_DIRS/autostart.
Nixos defines XDG_CONFIG_DIRS as /etc/xdg. So we need to copy the desktop files into the right folder
with make-startupitem.
This will override the existing winsymlinks setting. nativestrict
will cause ln to fail if it's unable to create a native symlink.
Native symlinks are required for the windows dll loader to find the
libraries.
This script is also used for cross-mingw, but setting CYGWIN
shouldn't cause a problem.
This allows users to specify a custom registry src,
because currently every packager would need to create
an outdated Cargo.lock just to be compatible with the
probably outdated rustRegistry in nixpkgs.
Currently there is no easy way to convince cargo to
do that, so this makes that workaround unnecessary.
I think it's ok to export things which aren't wrapped. The cc-wrapper
can be thought of as responsible for all of binutils and the c
compiler, only wrapping those binaries which are necessary to
interposition---as opposed to all binaries it thinks are relevaant.
Conversely, adding the setup hook to the unwrapped compilers would be
unforunate as hooks are ugly hacks and the compilers themselves take
a long time to rebuild. Better to wholely separate "pure packages" from
hacks.
Eventually we should avoid this "pre-wrapping" and just update those
files in nixpkgs. This mass-rebuild change is best done along with
those needed to reduce the disparity between native and cross (i.e.
making native the "identity cross").
We now (on cross) require per-target flag interposition by putting the
triple in the names of the relevant environment variables, e.g:
export NIX_arm_unknown_linux_gnu_CFLAGS_COMPILE=...
The wrapper also has a `infixSalt` attribute (and "_" prefixed and
suffixed variants) to assist downstream packages.
Note how that the dashes are replaced to keep the identifier valid.
Using names like this allows us to keep the settings for different
compilers seperate.
I think it might be even better to use names like `NIX_{BUILD,HOST}...`
using the platform's role rather than the platform itself, but this
would be more work as the previous stages' tools would have to be re-
wrapped to take on their new role. I therefore didn't do this for now,
but that route should be thoroughly explored in the future.
This fixes the Stack Clash issue rediscovered by Qualys. See
https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt
for more information on the topic, specifically section III.
We don't have the kernel mitigation available because it is a Grsecurity
feature which we don't support anymore. Other distributions like Gentoo
Hardened and Arch already have `-fstack-check` enabled by default.
See the Gentoo page on Stack Clash for more information on this solution:
https://wiki.gentoo.org/wiki/Hardened/Gentoo_Hardened_and_Stack_Clash
This unfortunately doesn't apply to clang because `-fstack-check` is a
noop there. Note that the GCC implementation also has problems that could
be exploited to circumvent these checks but it is still better than
keeping it disabled.
* ultrastardx-beta: init at 1.3.5
* libbass, libbass_fx: init at 24
* ultrastar-creator: init at 2017-04-12
* buildSupport/plugins.nix: add diffPlugins
Helper function to compare expected plugin lists to the found plugins.
* ultrastar-manager: init at 2017-05-24
The plugins are built in their own derivations, speeding up (re-)compilation.
The `diffPlugins` function from `beets` is reused to test for changes in the
plugin list on updates.
* beets: switch to diffPlugins
The function is basically just extracted for better reusability.
This value is require to get c++ std include path for libclang based tools (vim plugins in my case).
I currently extract it this with this rather command:
```
eval echo $(nix-instantiate --eval --expr 'with (import <nixpkgs>) {}; clang.default_cxx_stdlib_compile')
```
it did not trigger any recompilation on my system.
`SSL_CERT_FILE` has been replaced with `NIX_SSL_CERT_FILE`.
Before this change using `fetchdarcs` resulted in an error message like:
```
Identifying repository http://hub.darcs.net/scravy/easyplot inventory
darcs failed: Not a repository: http://hub.darcs.net/scravy/easyplot (Peer certificate cannot be authenticated with given CA certificates)
HINT: Do you have the right URI for the repository?
builder for ‘/nix/store/imyvcs6lvb5yva66krc5wk39931sam8v-fetchdarcs.drv’ failed with exit code 2
```
I was getting the following error building tide from Melpa:
nix-build -E '(import <nixpkgs> {}).emacs25WithPackages (p: [p.melpaPackages.tide])'
File tide-20170509.1134.tar is large (10.2M), really open? (y or n) Error reading from stdin
builder for ‘/nix/store/gs9ik7yf8iilsikkfing74i70m0diax3-emacs-tide-20170509.1134.drv’ failed with exit code 255
cannot build derivation ‘/nix/store/m3p080aani4rw82llp8nqk93cw2nvirk-emacs-with-packages-25.2.drv’: 1 dependencies couldn't be built
Solution was to disable the large file warning threshold when
installing packages.
This still causes some uncached rebuilds, but master(!) and staging
move too fast forward rebuild-wise, so Hydra might never catch up.
(There are also other occasional problems.)
Therefore I merge at this point where the rebuild isn't that bad.
We need to make sure that `$revs` ends with a space, since files must always
end with newlines. The previous code ignored the last entry in `$revs`, because
read already returns non-zero exit code for the last entry, as it does not end
with a space.
This change fixes several defects in the way `wrapGAppsHook` selected
the executable to wrap.
Previously, it would wrap any top-level files in the target `/bin` and
`/libexec` directories, including directories and non-executable
files. In addition, it failed to wrap files in subdirectories.
Now, it uses `find` to iterate over these directory hierarchies,
selecting only executable files for wrapping.
