nixpkgs/pkgs/development/ruby-modules/gem/default.nix
Farid Zakaria 4af8bc084a
buildRubyGem: fix to support bundler cmds (#104977)
The way in which Nixpks builds Ruby gems means that certain operations
by bundler *will not work*, namely `bundle install --redownload`.

According to the source the _cache/_ directory should have been kept,
however it seems through revisions to the file it has been purged.

Here was the comment from the original commit that introduced
buildRubyGem:
```
  # Note:
  #   We really do need to keep the $out/${ruby.gemPath}/cache.
  #   This is very important in order for many parts of RubyGems/Bundler to not blow up.
  #   See https://github.com/bundler/bundler/issues/3327
```

Why is the _cache_ directory needed?

Bundler and RubyGems uses the cache as a source of truth.
When bundler executes `bundler install --redownload`, any gems it
discovers in the _GEM_PATH_ it assums must have their _.gem_ file
present in the cache (unaware it was installed from Nix).

Rather than downloading the gem from RubyGems the bundler code forcibly
re-installs the gem from the cache directory instead and **fails** if it
does not exist.

I've opened https://github.com/rubygems/rubygems/issues/4088 to see if
this failure should be soft and not so explicit; or fallback to fetching
the gem from scratch.

Without this change the following is the error:
```bash
> [nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install --force
[DEPRECATED] The `--force` option has been renamed to `--redownload`
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.1.4
Installing hello-world 1.2.0
Bundler::GemNotFound: Could not find hello-world-1.2.0.gem for installation
An error occurred while installing hello-world (1.2.0), and Bundler
cannot continue.
Make sure that `gem install hello-world -v '1.2.0' --source
'https://rubygems.org/'` succeeds before bundling.
```

Wth the fix the following no woccurs:
```bash
[nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install --redownload
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/69wjlj4yirp48rv1q03zxgd4xvf0150d-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.1.4
Installing hello-world 1.2.0
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
```

```
[nix-shell:~/code/nix/playground/jruby-bundler-rake]$ ls -l /nix/store/cwl9n5073hqgpfhnw4wic13nrrgg9dn8-gem-env/lib/jruby/gems/2.5.0/cache/
total 8
lrwxrwxrwx 1 fmzakari primarygroup 102 Dec 31  1969 bundler-2.1.4.gem -> /nix/store/ifc8a0gsfkrhkv953rd4rz8bcspahi8y-bundler-2.1.4/lib/jruby/gems/2.5.0/cache/bundler-2.1.4.gem
lrwxrwxrwx 1 fmzakari primarygroup 110 Dec 31  1969 hello-world-1.2.0.gem -> /nix/store/xi9ln6n1mz2is5ppykjxqhhkpjq9zm6i-hello-world-1.2.0/lib/jruby/gems/2.5.0/cache/hello-world-1.2.0.gem
```

I have a minimal project that demonstrates this issue at https://github.com/fzakaria/jruby-bundler-nix-failure
2020-11-30 12:15:12 +01:00

243 lines
6.9 KiB
Nix

# This builds gems in a way that is compatible with bundler.
#
# Bundler installs gems from git sources _very_ differently from how RubyGems
# installes gem packages, though they both install gem packages similarly.
#
# We monkey-patch Bundler to remove any impurities and then drive its internals
# to install git gems.
#
# For the sake of simplicity, gem packages are installed with the standard `gem`
# program.
#
# Note that bundler does not support multiple prefixes; it assumes that all
# gems are installed in a common prefix, and has no support for specifying
# otherwise. Therefore, if you want to be able to use the resulting derivations
# with bundler, you need to create a symlink forrest first, which is what
# `bundlerEnv` does for you.
#
# Normal gem packages can be used outside of bundler; a binstub is created in
# $out/bin.
{ lib, fetchurl, fetchgit, makeWrapper, git, darwin
, ruby, bundler
} @ defs:
lib.makeOverridable (
{ name ? null
, gemName
, version ? null
, type ? "gem"
, document ? [] # e.g. [ "ri" "rdoc" ]
, platform ? "ruby"
, ruby ? defs.ruby
, stdenv ? ruby.stdenv
, namePrefix ? (let
rubyName = builtins.parseDrvName ruby.name;
in "${rubyName.name}${rubyName.version}-")
, buildInputs ? []
, meta ? {}
, patches ? []
, gemPath ? []
, dontStrip ? false
# Assume we don't have to build unless strictly necessary (e.g. the source is a
# git checkout).
# If you need to apply patches, make sure to set `dontBuild = false`;
, dontBuild ? true
, dontInstallManpages ? false
, propagatedBuildInputs ? []
, propagatedUserEnvPkgs ? []
, buildFlags ? []
, passthru ? {}
# bundler expects gems to be stored in the cache directory for certain actions
# such as `bundler install --redownload`.
# At the cost of increasing the store size, you can keep the gems to have closer
# alignment with what Bundler expects.
, keepGemCache ? false
, ...} @ attrs:
let
src = attrs.src or (
if type == "gem" then
fetchurl {
urls = map (
remote: "${remote}/gems/${gemName}-${version}.gem"
) (attrs.source.remotes or [ "https://rubygems.org" ]);
inherit (attrs.source) sha256;
}
else if type == "git" then
fetchgit {
inherit (attrs.source) url rev sha256 fetchSubmodules;
}
else if type == "url" then
fetchurl attrs.source
else
throw "buildRubyGem: don't know how to build a gem of type \"${type}\""
);
documentFlag =
if document == []
then "-N"
else "--document ${lib.concatStringsSep "," document}";
in
stdenv.mkDerivation ((builtins.removeAttrs attrs ["source"]) // {
inherit ruby;
inherit dontBuild;
inherit dontStrip;
inherit type;
buildInputs = [
ruby makeWrapper
] ++ lib.optionals (type == "git") [ git ]
++ lib.optionals (type != "gem") [ bundler ]
++ lib.optional stdenv.isDarwin darwin.libobjc
++ buildInputs;
#name = builtins.trace (attrs.name or "no attr.name" ) "${namePrefix}${gemName}-${version}";
name = attrs.name or "${namePrefix}${gemName}-${version}";
inherit src;
unpackPhase = attrs.unpackPhase or ''
runHook preUnpack
if [[ -f $src && $src == *.gem ]]; then
if [[ -z "''${dontBuild-}" ]]; then
# we won't know the name of the directory that RubyGems creates,
# so we'll just use a glob to find it and move it over.
gempkg="$src"
sourceRoot=source
gem unpack $gempkg --target=container
cp -r container/* $sourceRoot
rm -r container
# copy out the original gemspec, for convenience during patching /
# overrides.
gem specification $gempkg --ruby > original.gemspec
gemspec=$(readlink -f .)/original.gemspec
else
gempkg="$src"
fi
else
# Fall back to the original thing for everything else.
dontBuild=""
preUnpack="" postUnpack="" unpackPhase
fi
runHook postUnpack
'';
buildPhase = attrs.buildPhase or ''
runHook preBuild
if [[ "$type" == "gem" ]]; then
if [[ -z "$gemspec" ]]; then
gemspec="$(find . -name '*.gemspec')"
echo "found the following gemspecs:"
echo "$gemspec"
gemspec="$(echo "$gemspec" | head -n1)"
fi
exec 3>&1
output="$(gem build $gemspec | tee >(cat - >&3))"
exec 3>&-
gempkg=$(echo "$output" | grep -oP 'File: \K(.*)')
echo "gem package built: $gempkg"
elif [[ "$type" == "git" ]]; then
git init
# remove variations to improve the likelihood of a bit-reproducible output
rm -rf .git/logs/ .git/hooks/ .git/index .git/FETCH_HEAD .git/ORIG_HEAD .git/refs/remotes/origin/HEAD .git/config
# support `git ls-files`
git add .
fi
runHook postBuild
'';
# Note:
# We really do need to keep the $out/${ruby.gemPath}/cache.
# This is very important in order for many parts of RubyGems/Bundler to not blow up.
# See https://github.com/bundler/bundler/issues/3327
installPhase = attrs.installPhase or ''
runHook preInstall
export GEM_HOME=$out/${ruby.gemPath}
mkdir -p $GEM_HOME
echo "buildFlags: $buildFlags"
${lib.optionalString (type == "url") ''
ruby ${./nix-bundle-install.rb} \
"path" \
'${gemName}' \
'${version}' \
'${lib.escapeShellArgs buildFlags}'
''}
${lib.optionalString (type == "git") ''
ruby ${./nix-bundle-install.rb} \
"git" \
'${gemName}' \
'${version}' \
'${lib.escapeShellArgs buildFlags}' \
'${attrs.source.url}' \
'.' \
'${attrs.source.rev}'
''}
${lib.optionalString (type == "gem") ''
if [[ -z "$gempkg" ]]; then
echo "failure: \$gempkg path unspecified" 1>&2
exit 1
elif [[ ! -f "$gempkg" ]]; then
echo "failure: \$gempkg path invalid" 1>&2
exit 1
fi
gem install \
--local \
--force \
--http-proxy 'http://nodtd.invalid' \
--ignore-dependencies \
--install-dir "$GEM_HOME" \
--build-root '/' \
--backtrace \
--no-env-shebang \
${documentFlag} \
$gempkg $gemFlags -- $buildFlags
# looks like useless files which break build repeatability and consume space
pushd $out/${ruby.gemPath}
rm -fv doc/*/*/created.rid || true
rm -fv {gems/*/ext/*,extensions/*/*/*}/{Makefile,mkmf.log,gem_make.out} || true
${if keepGemCache then "" else "rm -fvr cache"}
popd
# write out metadata and binstubs
spec=$(echo $out/${ruby.gemPath}/specifications/*.gemspec)
ruby ${./gem-post-build.rb} "$spec"
''}
${lib.optionalString (!dontInstallManpages) ''
for section in {1..9}; do
mandir="$out/share/man/man$section"
find $out/lib \( -wholename "*/man/*.$section" -o -wholename "*/man/man$section/*.$section" \) \
-execdir mkdir -p $mandir \; -execdir cp '{}' $mandir \;
done
''}
runHook postInstall
'';
propagatedBuildInputs = gemPath ++ propagatedBuildInputs;
propagatedUserEnvPkgs = gemPath ++ propagatedUserEnvPkgs;
passthru = passthru // { isRubyGem = true; };
inherit meta;
})
)