4af8bc084a
The way in which Nixpks builds Ruby gems means that certain operations by bundler *will not work*, namely `bundle install --redownload`. According to the source the _cache/_ directory should have been kept, however it seems through revisions to the file it has been purged. Here was the comment from the original commit that introduced buildRubyGem: ``` # Note: # We really do need to keep the $out/${ruby.gemPath}/cache. # This is very important in order for many parts of RubyGems/Bundler to not blow up. # See https://github.com/bundler/bundler/issues/3327 ``` Why is the _cache_ directory needed? Bundler and RubyGems uses the cache as a source of truth. When bundler executes `bundler install --redownload`, any gems it discovers in the _GEM_PATH_ it assums must have their _.gem_ file present in the cache (unaware it was installed from Nix). Rather than downloading the gem from RubyGems the bundler code forcibly re-installs the gem from the cache directory instead and **fails** if it does not exist. I've opened https://github.com/rubygems/rubygems/issues/4088 to see if this failure should be soft and not so explicit; or fallback to fetching the gem from scratch. Without this change the following is the error: ```bash > [nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install --force [DEPRECATED] The `--force` option has been renamed to `--redownload` WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Fetching gem metadata from https://rubygems.org/. Using bundler 2.1.4 Installing hello-world 1.2.0 Bundler::GemNotFound: Could not find hello-world-1.2.0.gem for installation An error occurred while installing hello-world (1.2.0), and Bundler cannot continue. Make sure that `gem install hello-world -v '1.2.0' --source 'https://rubygems.org/'` succeeds before bundling. ``` Wth the fix the following no woccurs: ```bash [nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install --redownload WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/69wjlj4yirp48rv1q03zxgd4xvf0150d-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Fetching gem metadata from https://rubygems.org/. Using bundler 2.1.4 Installing hello-world 1.2.0 Bundle complete! 1 Gemfile dependency, 2 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. ``` ``` [nix-shell:~/code/nix/playground/jruby-bundler-rake]$ ls -l /nix/store/cwl9n5073hqgpfhnw4wic13nrrgg9dn8-gem-env/lib/jruby/gems/2.5.0/cache/ total 8 lrwxrwxrwx 1 fmzakari primarygroup 102 Dec 31 1969 bundler-2.1.4.gem -> /nix/store/ifc8a0gsfkrhkv953rd4rz8bcspahi8y-bundler-2.1.4/lib/jruby/gems/2.5.0/cache/bundler-2.1.4.gem lrwxrwxrwx 1 fmzakari primarygroup 110 Dec 31 1969 hello-world-1.2.0.gem -> /nix/store/xi9ln6n1mz2is5ppykjxqhhkpjq9zm6i-hello-world-1.2.0/lib/jruby/gems/2.5.0/cache/hello-world-1.2.0.gem ``` I have a minimal project that demonstrates this issue at https://github.com/fzakaria/jruby-bundler-nix-failure
243 lines
6.9 KiB
Nix
243 lines
6.9 KiB
Nix
# This builds gems in a way that is compatible with bundler.
|
|
#
|
|
# Bundler installs gems from git sources _very_ differently from how RubyGems
|
|
# installes gem packages, though they both install gem packages similarly.
|
|
#
|
|
# We monkey-patch Bundler to remove any impurities and then drive its internals
|
|
# to install git gems.
|
|
#
|
|
# For the sake of simplicity, gem packages are installed with the standard `gem`
|
|
# program.
|
|
#
|
|
# Note that bundler does not support multiple prefixes; it assumes that all
|
|
# gems are installed in a common prefix, and has no support for specifying
|
|
# otherwise. Therefore, if you want to be able to use the resulting derivations
|
|
# with bundler, you need to create a symlink forrest first, which is what
|
|
# `bundlerEnv` does for you.
|
|
#
|
|
# Normal gem packages can be used outside of bundler; a binstub is created in
|
|
# $out/bin.
|
|
|
|
{ lib, fetchurl, fetchgit, makeWrapper, git, darwin
|
|
, ruby, bundler
|
|
} @ defs:
|
|
|
|
lib.makeOverridable (
|
|
|
|
{ name ? null
|
|
, gemName
|
|
, version ? null
|
|
, type ? "gem"
|
|
, document ? [] # e.g. [ "ri" "rdoc" ]
|
|
, platform ? "ruby"
|
|
, ruby ? defs.ruby
|
|
, stdenv ? ruby.stdenv
|
|
, namePrefix ? (let
|
|
rubyName = builtins.parseDrvName ruby.name;
|
|
in "${rubyName.name}${rubyName.version}-")
|
|
, buildInputs ? []
|
|
, meta ? {}
|
|
, patches ? []
|
|
, gemPath ? []
|
|
, dontStrip ? false
|
|
# Assume we don't have to build unless strictly necessary (e.g. the source is a
|
|
# git checkout).
|
|
# If you need to apply patches, make sure to set `dontBuild = false`;
|
|
, dontBuild ? true
|
|
, dontInstallManpages ? false
|
|
, propagatedBuildInputs ? []
|
|
, propagatedUserEnvPkgs ? []
|
|
, buildFlags ? []
|
|
, passthru ? {}
|
|
# bundler expects gems to be stored in the cache directory for certain actions
|
|
# such as `bundler install --redownload`.
|
|
# At the cost of increasing the store size, you can keep the gems to have closer
|
|
# alignment with what Bundler expects.
|
|
, keepGemCache ? false
|
|
, ...} @ attrs:
|
|
|
|
let
|
|
src = attrs.src or (
|
|
if type == "gem" then
|
|
fetchurl {
|
|
urls = map (
|
|
remote: "${remote}/gems/${gemName}-${version}.gem"
|
|
) (attrs.source.remotes or [ "https://rubygems.org" ]);
|
|
inherit (attrs.source) sha256;
|
|
}
|
|
else if type == "git" then
|
|
fetchgit {
|
|
inherit (attrs.source) url rev sha256 fetchSubmodules;
|
|
}
|
|
else if type == "url" then
|
|
fetchurl attrs.source
|
|
else
|
|
throw "buildRubyGem: don't know how to build a gem of type \"${type}\""
|
|
);
|
|
documentFlag =
|
|
if document == []
|
|
then "-N"
|
|
else "--document ${lib.concatStringsSep "," document}";
|
|
|
|
in
|
|
|
|
stdenv.mkDerivation ((builtins.removeAttrs attrs ["source"]) // {
|
|
inherit ruby;
|
|
inherit dontBuild;
|
|
inherit dontStrip;
|
|
inherit type;
|
|
|
|
buildInputs = [
|
|
ruby makeWrapper
|
|
] ++ lib.optionals (type == "git") [ git ]
|
|
++ lib.optionals (type != "gem") [ bundler ]
|
|
++ lib.optional stdenv.isDarwin darwin.libobjc
|
|
++ buildInputs;
|
|
|
|
#name = builtins.trace (attrs.name or "no attr.name" ) "${namePrefix}${gemName}-${version}";
|
|
name = attrs.name or "${namePrefix}${gemName}-${version}";
|
|
|
|
inherit src;
|
|
|
|
|
|
unpackPhase = attrs.unpackPhase or ''
|
|
runHook preUnpack
|
|
|
|
if [[ -f $src && $src == *.gem ]]; then
|
|
if [[ -z "''${dontBuild-}" ]]; then
|
|
# we won't know the name of the directory that RubyGems creates,
|
|
# so we'll just use a glob to find it and move it over.
|
|
gempkg="$src"
|
|
sourceRoot=source
|
|
gem unpack $gempkg --target=container
|
|
cp -r container/* $sourceRoot
|
|
rm -r container
|
|
|
|
# copy out the original gemspec, for convenience during patching /
|
|
# overrides.
|
|
gem specification $gempkg --ruby > original.gemspec
|
|
gemspec=$(readlink -f .)/original.gemspec
|
|
else
|
|
gempkg="$src"
|
|
fi
|
|
else
|
|
# Fall back to the original thing for everything else.
|
|
dontBuild=""
|
|
preUnpack="" postUnpack="" unpackPhase
|
|
fi
|
|
|
|
runHook postUnpack
|
|
'';
|
|
|
|
buildPhase = attrs.buildPhase or ''
|
|
runHook preBuild
|
|
|
|
if [[ "$type" == "gem" ]]; then
|
|
if [[ -z "$gemspec" ]]; then
|
|
gemspec="$(find . -name '*.gemspec')"
|
|
echo "found the following gemspecs:"
|
|
echo "$gemspec"
|
|
gemspec="$(echo "$gemspec" | head -n1)"
|
|
fi
|
|
|
|
exec 3>&1
|
|
output="$(gem build $gemspec | tee >(cat - >&3))"
|
|
exec 3>&-
|
|
|
|
gempkg=$(echo "$output" | grep -oP 'File: \K(.*)')
|
|
|
|
echo "gem package built: $gempkg"
|
|
elif [[ "$type" == "git" ]]; then
|
|
git init
|
|
# remove variations to improve the likelihood of a bit-reproducible output
|
|
rm -rf .git/logs/ .git/hooks/ .git/index .git/FETCH_HEAD .git/ORIG_HEAD .git/refs/remotes/origin/HEAD .git/config
|
|
# support `git ls-files`
|
|
git add .
|
|
fi
|
|
|
|
runHook postBuild
|
|
'';
|
|
|
|
# Note:
|
|
# We really do need to keep the $out/${ruby.gemPath}/cache.
|
|
# This is very important in order for many parts of RubyGems/Bundler to not blow up.
|
|
# See https://github.com/bundler/bundler/issues/3327
|
|
installPhase = attrs.installPhase or ''
|
|
runHook preInstall
|
|
|
|
export GEM_HOME=$out/${ruby.gemPath}
|
|
mkdir -p $GEM_HOME
|
|
|
|
echo "buildFlags: $buildFlags"
|
|
|
|
${lib.optionalString (type == "url") ''
|
|
ruby ${./nix-bundle-install.rb} \
|
|
"path" \
|
|
'${gemName}' \
|
|
'${version}' \
|
|
'${lib.escapeShellArgs buildFlags}'
|
|
''}
|
|
${lib.optionalString (type == "git") ''
|
|
ruby ${./nix-bundle-install.rb} \
|
|
"git" \
|
|
'${gemName}' \
|
|
'${version}' \
|
|
'${lib.escapeShellArgs buildFlags}' \
|
|
'${attrs.source.url}' \
|
|
'.' \
|
|
'${attrs.source.rev}'
|
|
''}
|
|
|
|
${lib.optionalString (type == "gem") ''
|
|
if [[ -z "$gempkg" ]]; then
|
|
echo "failure: \$gempkg path unspecified" 1>&2
|
|
exit 1
|
|
elif [[ ! -f "$gempkg" ]]; then
|
|
echo "failure: \$gempkg path invalid" 1>&2
|
|
exit 1
|
|
fi
|
|
|
|
gem install \
|
|
--local \
|
|
--force \
|
|
--http-proxy 'http://nodtd.invalid' \
|
|
--ignore-dependencies \
|
|
--install-dir "$GEM_HOME" \
|
|
--build-root '/' \
|
|
--backtrace \
|
|
--no-env-shebang \
|
|
${documentFlag} \
|
|
$gempkg $gemFlags -- $buildFlags
|
|
|
|
# looks like useless files which break build repeatability and consume space
|
|
pushd $out/${ruby.gemPath}
|
|
rm -fv doc/*/*/created.rid || true
|
|
rm -fv {gems/*/ext/*,extensions/*/*/*}/{Makefile,mkmf.log,gem_make.out} || true
|
|
${if keepGemCache then "" else "rm -fvr cache"}
|
|
popd
|
|
|
|
# write out metadata and binstubs
|
|
spec=$(echo $out/${ruby.gemPath}/specifications/*.gemspec)
|
|
ruby ${./gem-post-build.rb} "$spec"
|
|
''}
|
|
|
|
${lib.optionalString (!dontInstallManpages) ''
|
|
for section in {1..9}; do
|
|
mandir="$out/share/man/man$section"
|
|
find $out/lib \( -wholename "*/man/*.$section" -o -wholename "*/man/man$section/*.$section" \) \
|
|
-execdir mkdir -p $mandir \; -execdir cp '{}' $mandir \;
|
|
done
|
|
''}
|
|
|
|
runHook postInstall
|
|
'';
|
|
|
|
propagatedBuildInputs = gemPath ++ propagatedBuildInputs;
|
|
propagatedUserEnvPkgs = gemPath ++ propagatedUserEnvPkgs;
|
|
|
|
passthru = passthru // { isRubyGem = true; };
|
|
inherit meta;
|
|
})
|
|
|
|
)
|