This is a major closure size reduction on Darwin, and probably a less
significant one on Linux. On darwin, retaining the compiler means adding
clang and its dependency llvm to the perl closure, which gives us ~400MB
of extra stuff. Considering that Nix itself depends on this version of
perl, that makes cutting a new Nix release rather unpleasaont Darwin.
After this patch, I was able to get the `nixUnstable` closure down to
21MB after feeding it into a .tar.xz (123MB before compression). There's
still room for improvement but this should carry us over until we split
outputs.
Some multiple-output changes were previously only in 5.22,
but since master is still using 5.20, let's stick with that version
on closure-size as well.
741bf840da (commitcomment-14784970)
http://hydra.nixos.org/eval/1234895
The mass errors on Hydra seem transient; I verified ghc on i686-linux.
Only darwin jobs are queued ATM. There's a libpng security update
included in this merge, so I don't want to wait too long.
This improves our Bundler integration (i.e. `bundlerEnv`).
Before describing the implementation differences, I'd like to point a
breaking change: buildRubyGem now expects `gemName` and `version` as
arguments, rather than a `name` attribute in the form of
"<gem-name>-<version>".
Now for the differences in implementation.
The previous implementation installed all gems at once in a single
derivation. This was made possible by using a set of monkey-patches to
prevent Bundler from downloading gems impurely, and to help Bundler
find and activate all required gems prior to installation. This had
several downsides:
* The patches were really hard to understand, and required subtle
interaction with the rest of the build environment.
* A single install failure would cause the entire derivation to fail.
The new implementation takes a different approach: we install gems into
separate derivations, and then present Bundler with a symlink forest
thereof. This has a couple benefits over the existing approach:
* Fewer patches are required, with less interplay with the rest of the
build environment.
* Changes to one gem no longer cause a rebuild of the entire dependency
graph.
* Builds take 20% less time (using gitlab as a reference).
It's unfortunate that we still have to muck with Bundler's internals,
though it's unavoidable with the way that Bundler is currently designed.
There are a number improvements that could be made in Bundler that would
simplify our packaging story:
* Bundler requires all installed gems reside within the same prefix
(GEM_HOME), unlike RubyGems which allows for multiple prefixes to
be specified through GEM_PATH. It would be ideal if Bundler allowed
for packages to be installed and sourced from multiple prefixes.
* Bundler installs git sources very differently from how RubyGems
installs gem packages, and, unlike RubyGems, it doesn't provide a
public interface (CLI or programmatic) to guide the installation of a
single gem. We are presented with the options of either
reimplementing a considerable portion Bundler, or patch and use parts
of its internals; I choose the latter. Ideally, there would be a way
to install gems from git sources in a manner similar to how we drive
`gem` to install gem packages.
* When a bundled program is executed (via `bundle exec` or a
binstub that does `require 'bundler/setup'`), the setup process reads
the Gemfile.lock, activates the dependencies, re-serializes the lock
file it read earlier, and then attempts to overwrite the Gemfile.lock
if the contents aren't bit-identical. I think the reasoning is that
by merely running an application with a newer version of Bundler, you'll
automatically keep the Gemfile.lock up-to-date with any changes in the
format. Unfortunately, that doesn't play well with any form of
packaging, because bundler will immediately cause the application to
abort when it attempts to write to the read-only Gemfile.lock in the
store. We work around this by normalizing the Gemfile.lock with the
version of Bundler that we'll use at runtime before we copy it into
the store. This feels fragile, but it's the best we can do without
changes upstream, or resorting to more delicate hacks.
With all of the challenges in using Bundler, one might wonder why we
can't just cut Bundler out of the picture and use RubyGems. After all,
Nix provides most of the isolation that Bundler is used for anyway.
The problem, however, is that almost every Rails application calls
`Bundler::require` at startup (by way of the default project templates).
Because bundler will then, by default, `require` each gem listed in the
Gemfile, Rails applications are almost always written such that none of
the source files explicitly require their dependencies. That leaves us
with two options: support and use Bundler, or maintain massive patches
for every Rails application that we package.
Closes#8612
Previously the gems defaulted to "ruby" as the name and
"${ruby-version}-${gem-name}-${gem-version}" as the version,
which was just insane.
https://github.com/NixOS/nixpkgs/issues/9771#issuecomment-141041414
Noone is reacting so it's high time to take at least some action.
/cc @cstrahan.
According to @zimbatm, he got the SHA256 hashes via nix-prefetch-git.
However, fetchFromGitHub doesn't use Git to fetch the sources but uses
fetchzip under the hood, so we get plain source directories in the Nix
store, which in turn are hashed.
Tested by:
nix-build --no-out-link -E 'map (x:
(builtins.getAttr x (import ./. {})).src
) [ "ruby_1_9_3" "ruby_2_0_0" "ruby_2_1_0" "ruby_2_1_1" "ruby_2_1_2"
"ruby_2_1_3" "ruby_2_1_6" "ruby_2_1_7" "ruby_2_2_0" "ruby_2_2_2"
"ruby_2_2_3"
]'
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Currently, when constructing a buildEnv and adding packages via
extraLibs, then binaries in extraLibs cannot access the other Python
modules. An example is having ipython/jupyter in extraLibs; in that case
ipython cannot import any other modules.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
The old way doesn't seem to work with multiple outputs.
TODO: this seems to be a recurring pattern with openssl, maybe the
splitting isn't done correctly?
Without this, python will silently build without native extensions (e.g
openssl); this causes wget to fail its tests for instance.
vcunat added python-3.5.
We were using HEAD for unreleased features. These features are now in
release builds so we should go back to using those. This also means we
won't have to deal with hash mismatches for all ruby packages.
This seems to have been confusing people, using both xlibs and xorg, etc.
- Avoided renaming local (and different) xlibs binding in gcc*.
- Fixed cases where both xorg and xlibs were used.
Hopefully everything still works as before.
Fixes#9044, close#9667. Thanks to @taku0 for suggesting this solution.
Now we have no modes starting with `/` or `+`.
Rewrite the `-perm` parameters of find:
- completely safe: rewrite `/0100` and `+100` to `-0100`,
- slightly semantics-changing: rewrite `+111` to `-0100`.
I cross-verified the `find` manual pages for Linux, Darwin, FreeBSD.
Darwin disallows shebang scripts from using other scripts as their
command--the command must be a binary. This commit changes the `mix`
shebang script from calling `elixir` directly (another shebang script),
instead using `env` as an intermediary.
Fixes#9050
This reverts commit cd52c04456 and
others.
Managing certificates (including revoking certificates and adding
custom certificates) becomes extremely painful if every package in the
system potentially depends on a different copy of cacert. Also, it
makes updating cacert rather expensive.
The first beta of php7 has been released. This enables developers to
test their code for breakage in preparation for the stable release and
allows us to make sure we don't have any bugs in the packaging prior to
the stable release.
