JakeHillion/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4e65797018
nixpkgs/pkgs/build-support/rust/fetchCargoTarball.nix

87 lines
2.6 KiB
Nix
Raw Normal View History

treewide: stdenv.lib -> lib
2021-01-24 00:40:18 +00:00
{ lib, stdenv, cacert, git, cargo, python3 }:
fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir This has several advantages: 1. It takes up less space on disk in-between builds in the nix store. 2. It uses less space in the binary cache for vendor derivation packages. 3. It uses less network traffic downloading from the binary cache. 4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only substitute --flat hashes on single files (not recursive directory hashes). 5. It's consistent with how simple `fetchurl` src derivations work. 6. It provides a stronger abstraction between input src-package and output package, e.g., it's harder to accidentally depend on the src derivation at runtime by referencing something like `${src}/etc/index.html`. Likewise, in the store it's harder to get confused with something that is just there as a build-time dependency vs. a runtime dependency, since the build-time src dependencies are tarred up. Disadvantages are: 1. It takes slightly longer to untar at the start of a build. As currently implemented, this attaches the compacted vendor.tar.gz feature as a rider on `verifyCargoDeps`, since both of them are relatively newly implemented behavior that change the `cargoSha256`. If this PR is accepted, I will push forward the remaining rust packages with a series of treewide PRs to update the `cargoSha256`s.
2020-01-12 16:21:23 +00:00
let cargo-vendor-normalise = stdenv.mkDerivation {
name = "cargo-vendor-normalise";
src = ./cargo-vendor-normalise.py;
nativeBuildInputs = [ python3.pkgs.wrapPython ];
dontUnpack = true;
installPhase = "install -D $src $out/bin/cargo-vendor-normalise";
pythonPath = [ python3.pkgs.toml ];
postFixup = "wrapPythonPrograms";
doInstallCheck = true;
installCheckPhase = ''
# check that ./fetchcargo-default-config.toml is a fix point
reference=${./fetchcargo-default-config.toml}
< $reference $out/bin/cargo-vendor-normalise > test;
cmp test $reference
'';
preferLocalBuild = true;
};
in
{ name ? "cargo-deps"
, src ? null
, srcs ? []
, patches ? []
fetchCargoTarball: set default sourceRoot to the empty string This avoids that non-buildRustPackage derivations need to specify sourceRoot when the fetcher performs root stripping.
2021-02-11 18:12:43 +00:00
, sourceRoot ? ""
buildRustPackage: add `cargoHash` for SRI hashes of vendored deps `buildRustPackage` currently accepts `cargoSha256` as a hash for vendored dependencies. This change adds `cargoHash` which accepts SRI hashes, setting `outputHashAlgo` to `null`. The hash mismatch message still uses `cargoSha256` as an example, which it probably should until we completely switch to SRI hashes.
2020-11-08 07:47:12 +00:00
, hash ? ""
, sha256 ? ""
fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir This has several advantages: 1. It takes up less space on disk in-between builds in the nix store. 2. It uses less space in the binary cache for vendor derivation packages. 3. It uses less network traffic downloading from the binary cache. 4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only substitute --flat hashes on single files (not recursive directory hashes). 5. It's consistent with how simple `fetchurl` src derivations work. 6. It provides a stronger abstraction between input src-package and output package, e.g., it's harder to accidentally depend on the src derivation at runtime by referencing something like `${src}/etc/index.html`. Likewise, in the store it's harder to get confused with something that is just there as a build-time dependency vs. a runtime dependency, since the build-time src dependencies are tarred up. Disadvantages are: 1. It takes slightly longer to untar at the start of a build. As currently implemented, this attaches the compacted vendor.tar.gz feature as a rider on `verifyCargoDeps`, since both of them are relatively newly implemented behavior that change the `cargoSha256`. If this PR is accepted, I will push forward the remaining rust packages with a series of treewide PRs to update the `cargoSha256`s.
2020-01-12 16:21:23 +00:00
, cargoUpdateHook ? ""
, ...
} @ args:
buildRustPackage: add `cargoHash` for SRI hashes of vendored deps `buildRustPackage` currently accepts `cargoSha256` as a hash for vendored dependencies. This change adds `cargoHash` which accepts SRI hashes, setting `outputHashAlgo` to `null`. The hash mismatch message still uses `cargoSha256` as an example, which it probably should until we completely switch to SRI hashes.
2020-11-08 07:47:12 +00:00
let hash_ =
if hash != "" then { outputHashAlgo = null; outputHash = hash; }
else if sha256 != "" then { outputHashAlgo = "sha256"; outputHash = sha256; }
else throw "fetchCargoTarball requires a hash for ${name}";
in stdenv.mkDerivation ({
fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir This has several advantages: 1. It takes up less space on disk in-between builds in the nix store. 2. It uses less space in the binary cache for vendor derivation packages. 3. It uses less network traffic downloading from the binary cache. 4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only substitute --flat hashes on single files (not recursive directory hashes). 5. It's consistent with how simple `fetchurl` src derivations work. 6. It provides a stronger abstraction between input src-package and output package, e.g., it's harder to accidentally depend on the src derivation at runtime by referencing something like `${src}/etc/index.html`. Likewise, in the store it's harder to get confused with something that is just there as a build-time dependency vs. a runtime dependency, since the build-time src dependencies are tarred up. Disadvantages are: 1. It takes slightly longer to untar at the start of a build. As currently implemented, this attaches the compacted vendor.tar.gz feature as a rider on `verifyCargoDeps`, since both of them are relatively newly implemented behavior that change the `cargoSha256`. If this PR is accepted, I will push forward the remaining rust packages with a series of treewide PRs to update the `cargoSha256`s.
2020-01-12 16:21:23 +00:00
name = "${name}-vendor.tar.gz";
nativeBuildInputs = [ cacert git cargo-vendor-normalise cargo ];
phases = "unpackPhase patchPhase buildPhase installPhase";
buildPhase = ''
# Ensure deterministic Cargo vendor builds
export SOURCE_DATE_EPOCH=1
if [[ ! -f Cargo.lock ]]; then
echo
echo "ERROR: The Cargo.lock file doesn't exist"
echo
buildRustPackage: add `cargoHash` for SRI hashes of vendored deps `buildRustPackage` currently accepts `cargoSha256` as a hash for vendored dependencies. This change adds `cargoHash` which accepts SRI hashes, setting `outputHashAlgo` to `null`. The hash mismatch message still uses `cargoSha256` as an example, which it probably should until we completely switch to SRI hashes.
2020-11-08 07:47:12 +00:00
echo "Cargo.lock is needed to make sure that cargoHash/cargoSha256 doesn't change"
fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir This has several advantages: 1. It takes up less space on disk in-between builds in the nix store. 2. It uses less space in the binary cache for vendor derivation packages. 3. It uses less network traffic downloading from the binary cache. 4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only substitute --flat hashes on single files (not recursive directory hashes). 5. It's consistent with how simple `fetchurl` src derivations work. 6. It provides a stronger abstraction between input src-package and output package, e.g., it's harder to accidentally depend on the src derivation at runtime by referencing something like `${src}/etc/index.html`. Likewise, in the store it's harder to get confused with something that is just there as a build-time dependency vs. a runtime dependency, since the build-time src dependencies are tarred up. Disadvantages are: 1. It takes slightly longer to untar at the start of a build. As currently implemented, this attaches the compacted vendor.tar.gz feature as a rider on `verifyCargoDeps`, since both of them are relatively newly implemented behavior that change the `cargoSha256`. If this PR is accepted, I will push forward the remaining rust packages with a series of treewide PRs to update the `cargoSha256`s.
2020-01-12 16:21:23 +00:00
echo "when the registry is updated."
echo
exit 1
fi
# Keep the original around for copyLockfile
cp Cargo.lock Cargo.lock.orig
export CARGO_HOME=$(mktemp -d cargo-home.XXX)
CARGO_CONFIG=$(mktemp cargo-config.XXXX)
${cargoUpdateHook}
cargo vendor $name | cargo-vendor-normalise > $CARGO_CONFIG
# Add the Cargo.lock to allow hash invalidation
cp Cargo.lock.orig $name/Cargo.lock
# Packages with git dependencies generate non-default cargo configs, so
# always install it rather than trying to write a standard default template.
install -D $CARGO_CONFIG $name/.cargo/config;
'';
# Build a reproducible tar, per instructions at https://reproducible-builds.org/docs/archives/
installPhase = ''
tar --owner=0 --group=0 --numeric-owner --format=gnu \
--sort=name --mtime="@$SOURCE_DATE_EPOCH" \
-czf $out $name
'';
buildRustPackage: add `cargoHash` for SRI hashes of vendored deps `buildRustPackage` currently accepts `cargoSha256` as a hash for vendored dependencies. This change adds `cargoHash` which accepts SRI hashes, setting `outputHashAlgo` to `null`. The hash mismatch message still uses `cargoSha256` as an example, which it probably should until we completely switch to SRI hashes.
2020-11-08 07:47:12 +00:00
inherit (hash_) outputHashAlgo outputHash;
fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir This has several advantages: 1. It takes up less space on disk in-between builds in the nix store. 2. It uses less space in the binary cache for vendor derivation packages. 3. It uses less network traffic downloading from the binary cache. 4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only substitute --flat hashes on single files (not recursive directory hashes). 5. It's consistent with how simple `fetchurl` src derivations work. 6. It provides a stronger abstraction between input src-package and output package, e.g., it's harder to accidentally depend on the src derivation at runtime by referencing something like `${src}/etc/index.html`. Likewise, in the store it's harder to get confused with something that is just there as a build-time dependency vs. a runtime dependency, since the build-time src dependencies are tarred up. Disadvantages are: 1. It takes slightly longer to untar at the start of a build. As currently implemented, this attaches the compacted vendor.tar.gz feature as a rider on `verifyCargoDeps`, since both of them are relatively newly implemented behavior that change the `cargoSha256`. If this PR is accepted, I will push forward the remaining rust packages with a series of treewide PRs to update the `cargoSha256`s.
2020-01-12 16:21:23 +00:00
treewide: stdenv.lib -> lib
2021-01-24 00:40:18 +00:00
impureEnvVars = lib.fetchers.proxyImpureEnvVars;
fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir This has several advantages: 1. It takes up less space on disk in-between builds in the nix store. 2. It uses less space in the binary cache for vendor derivation packages. 3. It uses less network traffic downloading from the binary cache. 4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only substitute --flat hashes on single files (not recursive directory hashes). 5. It's consistent with how simple `fetchurl` src derivations work. 6. It provides a stronger abstraction between input src-package and output package, e.g., it's harder to accidentally depend on the src derivation at runtime by referencing something like `${src}/etc/index.html`. Likewise, in the store it's harder to get confused with something that is just there as a build-time dependency vs. a runtime dependency, since the build-time src dependencies are tarred up. Disadvantages are: 1. It takes slightly longer to untar at the start of a build. As currently implemented, this attaches the compacted vendor.tar.gz feature as a rider on `verifyCargoDeps`, since both of them are relatively newly implemented behavior that change the `cargoSha256`. If this PR is accepted, I will push forward the remaining rust packages with a series of treewide PRs to update the `cargoSha256`s.
2020-01-12 16:21:23 +00:00
} // (builtins.removeAttrs args [
"name" "sha256" "cargoUpdateHook"
]))
Reference in New Issue Copy Permalink