nixpkgs/pkgs/build-support/fetchzip/default.nix

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

80 lines
2.3 KiB
Nix
Raw Normal View History

# This function downloads and unpacks an archive file, such as a zip
# or tar file. This is primarily useful for dynamically generated
# archives, such as GitHub's /archive URLs, where the unpacked content
# of the zip file doesn't change, but the zip file itself may
# (e.g. due to minor changes in the compression algorithm, or changes
# in timestamps).
{ lib, fetchurl, unzip, glibcLocalesUtf8 }:
{ # Optionally move the contents of the unpacked tree up one level.
stripRoot ? true
, url ? ""
, urls ? []
, extraPostFetch ? ""
, postFetch ? ""
, name ? "source"
, nativeBuildInputs ? [ ]
fetchzip: add extension parameter fetchzip downloads the file from specified URL, renames it to basename of that url, and then relies on unzip to do the unpacking. The first consequence is that this requires URL to end with proper extension—otherwise it will fail to unpack. This is not always the case and input-fonts workarounds this by adding “&.zip” query parameter (which is obviously a hack and is not guaranteed to work with every URL). The second consequence is that basename of the url must be a valid filename. I’ve tried to build a custom configuration of input-fonts and I get an error from mv that the filename is too long: > trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 406k 100 406k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k > mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long We could use “name” parameter as the filename (that’s how it is used in fetchurl). However, the previous attempt to do so (fc01353703426458d6990239936659ed3130d294) was reverted (24b5eb61eb944036b5716788ead1b204fba08719) because of the introduced regression—many fetchzip invocations use names without extension (also the default name is just “source”). This commit adds an optional “extension” parameter. If it is set, fetchzip renames the downloaded file to “download.${extension}” effectively solving both problems above without introducing a massive regression. This is a no-op for all existing packages. Tested by updating my NixOS setup + the extra inputs-fonts configuration mentioned above + tons of unstable emacs packages after a nix-collect-garbage (3Gb downloaded) with this patch applied.
2021-08-10 09:21:27 +01:00
, # Allows to set the extension for the intermediate downloaded
# file. This can be used as a hint for the unpackCmdHooks to select
# an appropriate unpacking tool.
extension ? null
, ... } @ args:
lib.warnIf (extraPostFetch != "") "use 'postFetch' instead of 'extraPostFetch' with 'fetchzip' and 'fetchFromGitHub'."
(fetchurl (let
fetchzip: add extension parameter fetchzip downloads the file from specified URL, renames it to basename of that url, and then relies on unzip to do the unpacking. The first consequence is that this requires URL to end with proper extension—otherwise it will fail to unpack. This is not always the case and input-fonts workarounds this by adding “&.zip” query parameter (which is obviously a hack and is not guaranteed to work with every URL). The second consequence is that basename of the url must be a valid filename. I’ve tried to build a custom configuration of input-fonts and I get an error from mv that the filename is too long: > trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 406k 100 406k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k > mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long We could use “name” parameter as the filename (that’s how it is used in fetchurl). However, the previous attempt to do so (fc01353703426458d6990239936659ed3130d294) was reverted (24b5eb61eb944036b5716788ead1b204fba08719) because of the introduced regression—many fetchzip invocations use names without extension (also the default name is just “source”). This commit adds an optional “extension” parameter. If it is set, fetchzip renames the downloaded file to “download.${extension}” effectively solving both problems above without introducing a massive regression. This is a no-op for all existing packages. Tested by updating my NixOS setup + the extra inputs-fonts configuration mentioned above + tons of unstable emacs packages after a nix-collect-garbage (3Gb downloaded) with this patch applied.
2021-08-10 09:21:27 +01:00
tmpFilename =
if extension != null
then "download.${extension}"
else baseNameOf (if url != "" then url else builtins.head urls);
in {
inherit name;
recursiveHash = true;
downloadToTemp = true;
# Have to pull in glibcLocalesUtf8 for unzip in setup-hook.sh to handle
# UTF-8 aware locale:
# https://github.com/NixOS/nixpkgs/issues/176225#issuecomment-1146617263
nativeBuildInputs = [ unzip glibcLocalesUtf8 ] ++ nativeBuildInputs;
postFetch =
''
unpackDir="$TMPDIR/unpack"
mkdir "$unpackDir"
cd "$unpackDir"
fetchzip: add extension parameter fetchzip downloads the file from specified URL, renames it to basename of that url, and then relies on unzip to do the unpacking. The first consequence is that this requires URL to end with proper extension—otherwise it will fail to unpack. This is not always the case and input-fonts workarounds this by adding “&.zip” query parameter (which is obviously a hack and is not guaranteed to work with every URL). The second consequence is that basename of the url must be a valid filename. I’ve tried to build a custom configuration of input-fonts and I get an error from mv that the filename is too long: > trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 406k 100 406k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k > mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long We could use “name” parameter as the filename (that’s how it is used in fetchurl). However, the previous attempt to do so (fc01353703426458d6990239936659ed3130d294) was reverted (24b5eb61eb944036b5716788ead1b204fba08719) because of the introduced regression—many fetchzip invocations use names without extension (also the default name is just “source”). This commit adds an optional “extension” parameter. If it is set, fetchzip renames the downloaded file to “download.${extension}” effectively solving both problems above without introducing a massive regression. This is a no-op for all existing packages. Tested by updating my NixOS setup + the extra inputs-fonts configuration mentioned above + tons of unstable emacs packages after a nix-collect-garbage (3Gb downloaded) with this patch applied.
2021-08-10 09:21:27 +01:00
renamed="$TMPDIR/${tmpFilename}"
mv "$downloadedFile" "$renamed"
unpackFile "$renamed"
chmod -R +w "$unpackDir"
''
+ (if stripRoot then ''
if [ $(ls "$unpackDir" | wc -l) != 1 ]; then
echo "error: zip file must contain a single file or directory."
echo "hint: Pass stripRoot=false; to fetchzip to assume flat list of files."
exit 1
fi
fn=$(cd "$unpackDir" && echo *)
if [ -f "$unpackDir/$fn" ]; then
mkdir $out
fi
mv "$unpackDir/$fn" "$out"
'' else ''
mv "$unpackDir" "$out"
'')
+ ''
${postFetch}
'' + ''
${extraPostFetch}
''
# Remove non-owner write permissions
# Fixes https://github.com/NixOS/nixpkgs/issues/38649
+ ''
chmod 755 "$out"
'';
} // removeAttrs args [ "stripRoot" "extraPostFetch" "postFetch" "extension" "nativeBuildInputs" ]))