Commit Graph

3643 Commits

Author SHA1 Message Date
Jan Tojnar
477ea066ca
Merge pull request #80091 from NixOS/npregit-dumpall
nix-prefetch-git: report deepClone & leaveDotGit
2020-02-19 07:37:11 +01:00
Profpatsch
493c357720 pkgs/build-support/writers: use runCommandLocal for scripts
If we just want to write a non-compiled script (e.g. writeDash), it’s
usually a lot faster just doing it locally. That’s what
`runCommandLocal` was introduced for, so let’s use it in `writers`.
2020-02-18 19:53:18 +01:00
(cdep)illabout
55e1529335 symlinkJoin: add more documentation 2020-02-18 16:35:28 +01:00
(cdep)illabout
ae607de23d linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a list of derivations 2020-02-18 16:35:28 +01:00
Andreas Rammhold
453589696b
Merge pull request #79816 from andir/buildRustCrate-no-override-dep
buildRustCrate: remove superfluous dependency overrides
2020-02-18 15:18:44 +01:00
Bas van Dijk
4e0a6bc4be symlinkJoin: fix "bash: Argument list too long" error
When the `paths` argument is too big `symlinkJoin` will fail with:

```
while setting up the build environment: executing '/nix/store/rm1hz1lybxangc8sdl7xvzs5dcvigvf7-bash-4.4-p23/bin/bash': Argument list too long
```

This is fixed by passing `paths` as a file instead of as an
environment variable.
2020-02-17 11:45:44 +01:00
Benjamin Hipple
6d881472ef rust: Fix for legacy fetch cargo
See inline comment and #79975 for details.
2020-02-15 17:41:35 -08:00
Jan Tojnar
28faf5bc86
nix-prefetch-git: Fix output mangling
`git repack` and `git gc` sometimes print “Nothing new to pack.”
to stdout, which breaks redirecting output to JSON file.

Let’s move the stdout of all git calls where it is not used to stderr
so that we still receive the info but it does not pollute our output.
2020-02-14 22:15:36 +01:00
Bjørn Forsman
d394c82cde
build-fhs-userenv: runCommand -> runCommandLocal (#77253)
It's a trivial derivation, no need to build remotely (it's slower that
way).
2020-02-14 20:16:49 +02:00
lewo
d49804cf45
Merge pull request #80102 from nlewo/fix-78744
dockerTools.buildLayeredImage: store all paths passed in final layer (fix 78744)
2020-02-14 09:48:52 +01:00
Mario Rodas
7696369bec
docker-slim: move expression to outside of build-support (#80078) 2020-02-14 09:42:18 +01:00
Antoine Eiche
baa78de594 nixosTests.docker-tools: add bulk-layer test
A regression test for https://github.com/NixOS/nixpkgs/issues/78744.
2020-02-14 09:26:26 +01:00
Richard Wallace
3b65b3f6d6 dockerTools.buildLayeredImage: store all paths passed in final layer
Fixes #78744

My previous change broke when there are more packages than the maximum
number of layers. I had assumed that the `store-path-to-layer.sh` was
only ever passed a single store path, but that is not the case if
there are multiple packages going into the final layer. To fix this, we
loop through the paths going into the final layer, appending them to the
tar file and making sure they end up at the right path.
2020-02-14 09:26:26 +01:00
Benjamin Hipple
eb11feaa0b treewide: change fetchCargoTarball default to opt-out
Changes the default fetcher in the Rust Platform to be the newer
`fetchCargoTarball`, and changes every application using the current default to
instead opt out.

This commit does not change any hashes or cause any rebuilds. Once integrated,
we will start deleting the opt-outs and recomputing hashes.

See #79975 for details.
2020-02-13 22:41:37 -08:00
Jan Tojnar
f19a101a73
nix-prefetch-git: report deepClone & leaveDotGit
Just like in be30ba8e0e
we want `fetchgit (builtins.fromJSON (builtins.readFile ./src.json))` to work.
2020-02-14 06:05:43 +01:00
Benjamin Hipple
c1542fc4cd rust: Replace migration README with GitHub issue
The readme was nice to discuss in the implementation PR, but now that this is
merged it's better to have an issue that can be linked against in PRs and
doesn't require further merges to update status.

Ported with a status update in #79975
2020-02-12 22:28:53 -05:00
Frederik Rietdijk
424697d512 Merge master into staging-next 2020-02-12 09:55:31 +01:00
Andreas Rammhold
be5597fc9d
buildRustCrate: remove superfluous dependency overrides
By overriding each dependency on every level of the dependency tree we
are creating a lot of unnecessary instances of the same derivation

Looking at the output size of `nix-instantiate --trace-function-calls
-vvvv …` and the execution time I got about a 10x improvement after
applying this change.

It was probably good intentions that lead to these overrides but in
practice no tooling (that I know of) really needs this. `carnix` and
`crate2nix` are fine without those overrides. Furthermore I believe that
it is the job of the tooling around `buildRustCrate` to provide a
coherent set of overrides. By not enforcing all of the overrides, debug
flags, verbosity, … to be the same throughout the closure we also allow
consumers to override specific aspects of the crates. Some (older?)
crates might need different `crateOverrides` then newer crates with the
same name. Currently such situations can not (easily) be implemented
with the override in-place.
2020-02-11 11:48:45 +01:00
Benjamin Hipple
2115a2037c fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir
This has several advantages:

1. It takes up less space on disk in-between builds in the nix store.
2. It uses less space in the binary cache for vendor derivation packages.
3. It uses less network traffic downloading from the binary cache.
4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only
   substitute --flat hashes on single files (not recursive directory hashes).
5. It's consistent with how simple `fetchurl` src derivations work.
6. It provides a stronger abstraction between input src-package and output
   package, e.g., it's harder to accidentally depend on the src derivation at
   runtime by referencing something like `${src}/etc/index.html`. Likewise, in
   the store it's harder to get confused with something that is just there as a
   build-time dependency vs. a runtime dependency, since the build-time
   src dependencies are tarred up.

Disadvantages are:
1. It takes slightly longer to untar at the start of a build.

As currently implemented, this attaches the compacted vendor.tar.gz feature as a
rider on `verifyCargoDeps`, since both of them are relatively newly implemented
behavior that change the `cargoSha256`.

If this PR is accepted, I will push forward the remaining rust packages with a
series of treewide PRs to update the `cargoSha256`s.
2020-02-10 10:17:29 -05:00
Andreas Rammhold
56e11bc8df
buildRustCrate: remap the current build dir to / for (more) reproducible builds 2020-02-06 01:18:59 +01:00
Uri Baghin
10d1a04b02 bazel-deps: remove 2020-02-04 12:35:47 +11:00
Uri Baghin
3dc9c1b885 bazel-remote: 2019-01-12 -> 2020-01-29 2020-02-04 02:26:15 +01:00
worldofpeace
8ecbde9def
Merge pull request #78945 from jbedo/singularity-img
singularity-tools: fix buildImage
2020-02-03 15:19:47 -05:00
Justin Bedo
56c22350c0
singularity-tools: fix buildImage 2020-02-02 09:18:39 +11:00
Silvan Mosberger
0a351c3f65
dockerTools.*: Assertion against building for Darwin (#77952)
dockerTools.*: Assertion against building for Darwin
2020-01-31 21:17:40 +01:00
lewo
86f8732194
Merge pull request #78834 from nlewo/fix-two-layers-image
Fix dockerTools.buildLayerImage with 2 layers
2020-01-30 21:10:52 +01:00
Antoine Eiche
01a68479cc dockerTools.buildLayeredImage: assert maxLayers > 1
Since a layer is reserved for "customization", the image can not
contains less than 2 layers.

The user gets the following message at evaluation:

    nix-instantiate nixos/tests/docker-tools.nix
    trace: the maxLayers argument of dockerTools.buildLayeredImage function must be greather than 1 (current value: 1)
2020-01-30 21:09:52 +01:00
Antoine Eiche
283bcc1003 dockerTools.buildLayeredImage: fix image with only 2 layers
A test is also added to ensure an image with 2 layers can be built.
2020-01-30 21:09:44 +01:00
Andreas Rammhold
a57d0fe0bb
buildRustCrate: fix #78412
`build.rs` files might create files. Those files are supposed to go into
`OUT_DIR` (envirionment variable) and not be overlayed onto the source
tree.
2020-01-28 14:07:58 +01:00
Andreas Rammhold
19698d15ce
buildRustCrateTests: add regression test for #74071 2020-01-28 14:07:58 +01:00
Frederik Rietdijk
7562c2112d
Merge pull request #77739 from NixOS/staging-next
Staging next
2020-01-28 14:03:29 +01:00
Andreas Rammhold
cd27940df0
Merge pull request #78188 from andir/buildRustCrate-support-rlib
buildRustCrate: support `rlib` dependency type
2020-01-28 13:47:57 +01:00
Frederik Rietdijk
dce0ca29d9 Merge master into staging-next 2020-01-28 10:46:13 +01:00
Mario Rodas
c3ccac15eb
libredirect: fix build on darwin
Fix build failure on darwin due to absence of `O_TMPFILE`.
2020-01-23 20:20:20 -05:00
Andreas Rammhold
78faab1be0 buildRustCrateTests: add test case for rlib linking 2020-01-21 17:46:32 +01:00
Andreas Rammhold
406e0c9d51
buildRustCrateTests: fix some formatting issues 2020-01-21 17:32:48 +01:00
Andreas Rammhold
d6a8b55fb0
buildRustCrate: treat rlib crates just like lib crates
Both version provide `rlib` files to link against. Previously we would
try to find a matching shared library in the `lib` output.
2020-01-21 17:22:59 +01:00
Andreas Rammhold
60fea726de
Merge pull request #77822 from andir/buildRustCrate-remove-custom-lib-path
buildRustCrate: remove custom lib path heuristic
2020-01-21 17:21:55 +01:00
Silvan Mosberger
b6a9211bf4
dockerTools.*: Assertion against building for darwin
Building a docker image with darwin binaries just yields a confusing
error when ran:

  standard_init_linux.go:211: exec user process caused "exec format error"

This change prevents people from building such images in the first place
2020-01-20 19:14:12 +01:00
Jonas Schürmann
61870727de mirrors.nix: Use HTTPS in maven mirrors
Since 2020-01-15, the maven central repository does no longer support HTTP.
See https://blog.sonatype.com/central-repository-moving-to-https for details.
2020-01-20 09:54:14 +01:00
Mario Rodas
21d0b2067f
Merge pull request #76831 from mbrgm/docker-slim
docker-slim: init at 1.26.1
2020-01-18 07:51:03 -05:00
Andreas Rammhold
69c96adc53
buildRustCrateTests: use releaseTools.aggregate
Previously I did use `runCommand` to do the same. Using
releaseTools.aggregate seems a lot saner and we might get nicer hydra
output of the tests that are failing.
2020-01-16 13:24:15 +01:00
Andreas Rammhold
29a8575e3d
buildRustCrate: remove one of the odd library filename cases
It used to be the case (ref missing) that cargo did treat
`src/$libName.rs` as an alternative to `src/lib.rs` when the latter
wasn't present. Recently I failed to reproduce that with vanilla cargo
and it started to cause pain with some crates of the form:

some_crate/
 `- src
   `- main.rs
   `- some_crate.rs

We would build `src/some_crate.rs` and thing it is a library while that
might not be the actual case. This crate is a valid `bin` crate not a
`lib` crate as far as I can tell from the samples I took.

I removed support for the previously required heuristic and commented
out the test cases in case we will need them again. We could crawl in
the Git history but chances are that the next person looking into this
doesn't know about the history.
2020-01-16 13:24:13 +01:00
Anders Kaseorg
3cd8ce3bce treewide: Fix unsafe concatenation of $LD_LIBRARY_PATH
Naive concatenation of $LD_LIBRARY_PATH can result in an empty
colon-delimited segment; this tells glibc to load libraries from the
current directory, which is definitely wrong, and may be a security
vulnerability if the current directory is untrusted.  (See #67234, for
example.)  Fix this throughout the tree.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-01-15 09:47:03 +01:00
Andreas Rammhold
cb007e69a1 stdenv: make symlinks that refer to the same output relative
While looking at the graph of all the outputs in my personal binary
cache it became obvious that we have a lot of self references within the
package set. That isn't an isuse by itself. However it increases the
size of the binary cache for every (reproducible) build of a package
that carries references to itself. You can no longer deduplicate the
outputs since they are all unique. One of the ways to get rid of (a few)
references is to rewrite all the symlinks that are currently used to be
relative symlinks. Two build of something that didn't really change but
carries a self-reference can the be store as the same NAR file again.

I quickly hacked together this change to see if that would yield and
success. My bash scripting skills are probably not great but so far it
seem to somewhat work.
2020-01-15 09:26:40 +01:00
Benjamin Hipple
6e8c377562 rustPlatform.buildRustPackage: cleaner output on verifyCargoDeps (#77567)
When this fails, the user may want to copy-paste the path to the "bad"
Cargo.lock file to inspect. The trailing `.` on `$cargoDeps.` gets caught in
most terminal copy-pastes. Since half the lines already don't have it, this
removes it from all of them for consistent output.
2020-01-12 17:19:17 +00:00
Antoine Eiche
da261e3631 dockerTools.buildLayeredImage: fix typo in comments 2020-01-11 09:02:30 +01:00
lewo
0d983f9f99
Merge pull request #75911 from Simspace/parallel-docker-buildlayeredimage
when building a layered docker image, ignore it if tar encounters cha…
2020-01-11 08:55:34 +01:00
Andreas Rammhold
3e61906e1c
buildRustCrate: slight "rewording" and reformatting
There is no point in reinventinb builtins through `filterAttrs` or the
like. Lets just stick to what we already have in our toolbelt.
2020-01-07 11:57:34 +01:00
Andreas Rammhold
a3a51763f9
buildRustCrate: add buildTests flag to tell rustc to build tests instead of binaries
This helps us instruct rustc to build tests instead of binaries. The
actual build will then ONLY produce test executables. This is a first
step towards having rust crate tests within nixpkgs.

We default back to only a single output in test cases since that is the
only reasonable thing to do here.

Producing libraries or binaries in addition to tests would theoretically
be feasible but usually generates different dependency trees. It is very
common to have some libraries in `[dev-depdendencies]` within Cargo.toml
just for your tests. To not start mixing things up going with a
dedicated derivation for the test build sounds like the best choice for
now.

To use this you must provide a proper test dependency chain to
`buildRustCrate` (as you would usually do with your non-test inputs).
And then set the `buildTests` attribute to `true`. The derivation will
then contain all tests that were built in `$out/tests`. All common test
patterns and directories should be supported and tested by this change.

Below is an example how you would run a single test from the derivation.
This commit contains some more examples in the `buildRustCrateTests`
attribute set that might be helpful.

```
let
  drv = buildRustCrate {
     …
     buildTests true;
  };
in runCommand "test-my-crate" {} ''
  touch $out
  exec ${drv}/tests/my-test
''
```
2020-01-07 11:57:34 +01:00