Without this, the generated pack files are non-deterministic.
I didn't notice this issue in my earlier testing, because my test repo
had too few commits for the thread scheduling to take effect. (Test repo
had about 10 commits.)
Add more files to the delete list:
* .git/FETCH_HEAD
* .git/ORIG_HEAD
* .git/refs/remotes/origin/HEAD
* .git/config
Further, remove all remote branches, remove tags not reachable from the
given 'rev', do a full repack and then garbage collect unreferenced
objects.
According to my testing, the result is fully deterministic. As in "any
change done to the upstream repo, ahead of 'rev', will not affect the
hash of the resulting 'clone'". Even changing the clone URL will not
change the output hash, because .git/config is removed.
A new version of git can of course change store format, but that's
unavoidable.
For big repositories, the repack operation may be a bit heavy. But as
far as I can see there is no cheaper way to determinism.
Hydra generates a GHC closure for Darwin that for no apparent reason
contains an ancient, broken Haddock binary -- probably because of an
impurity in the build system. That bug makes those GHC binaries
unusable: <https://github.com/NixOS/nixpkgs/issues/2689>.
This likely exacerbates the non-determinism in ghc package ids, so until
that is fixed let's live with the slow builds.
This reverts commit 817c0e4144.
This should fix the OpenJDK build, which was failing because paxctl is
in sbin and therefore not automatically added to $PATH.
http://hydra.nixos.org/build/15658346
This patch makes two changes.
(1) It memoizes the computation of dependsOnOld.
(2) It replaces rewrittenDerivations with a similar memoized table rewriteMemo.
This prevents the entire tree of run-time dependencies from being traversed and instead only traverses the graph of run-time dependencies.
In the case of deep dependency changes (such as changing one's bash version for an entire NixOS system) this can lead to an exponential speedup in processing time
because shared dependencies are no longer traversed multiple times.
This patch isn't quite derivation-per-derivation equivalent to the original computation.
There are two immaterial differences.
(1) The previous version would always call upon sed to replace oldDependency with newDependency even when the store object being updated doesn't directly depend on
oldDependency.
The new version only replaceds oldDependency with newDependency when the store object being updated actually directly depends on oldDependency (which means there is
actually a hash to replace).
(2) The previous version would list the old store object as a source input of the new store object, *except* for the root derivation being updated. Because the
root derivation being updated has its actual derivation avaiable the previous verions would make the updated root derivation depend on the old derivation as a
derivation input instead of a source input.
The new version always lists the old store object as a source input, including the root derivation.
We still need this because some clang-based packages depend on
it. (The sysroot filtering was originally done by clang-wrapper's
ld-wrapper, but we merged the ld-wrappers in
a4f9b9c8b5ec9ef106671ffdf93e0059835d0ec1.)
http://hydra.nixos.org/build/13906922
Fixes a regression on OS X introduced by f83af95.
Don't use --tmpdir for mktemp, because that flag doesn't exist on OS X.
However, using -t is deprecated in GNU coreutils, so as suggested by
@ip1981 we're now using parameter expansion on ${TMPDIR:-/tmp} to
provide /tmp as a fallback if TMPDIR is not set and use it instead.
Also use this approach for nix-prefetch-cvs now in order to stay
consistent.
Reported-by: Vladimir Kirillov <proger@wilab.org.ua>
Tested-by: Igor Pashev <pashev.igor@gmail.com>
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Instead of relying on $$ to not collide with an existing path.
Quoting the Bash manual about $$:
> Expands to the process ID of the shell. In a () subshell, it expands
> to the process ID of the current shell, not the subshell.
So, this is different from $BASHPID:
> Expands to the process ID of the current bash process. This differs
> from $$ under certain circumstances, such as subshells that do not
> require bash to be re-initialized.
But even $BASHPID is prone to race conditions if the process IDs wrap
around, so to be on the safe side, we're using mktemp here.
Closes#3784.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
For now, we don't NATIVE_SYSTEM_HEADER_DIR because it breaks the
build. However, it points to Glibc in the Nix store (not /usr/include)
so it's kind of okay.
Now gcc is just another build input, making it possible in the future
to have a stdenv that doesn't depend on a C compiler. This is very
useful on NixOS, since it would allow trivial builders like
writeTextFile to work without pulling in the C compiler.
If $src refers to a directory, then always copy it. Previously, we
checked the extension first, so if the directory had an extension like
.tar, unpackPhase would fail.
There was a few files containing timestamp, so we now remove them.
It shouldn't be a problem for logs. However, index might be. Anyway,
that's better than nothing.
Having a separate clang-wrapper is really unfortunate because it
means that we'll forever forget to apply changes to both (e.g.
commit 289895fe2c). This commit
gets rid of the redundant copies of ld-wrapper.sh and utils.sh.
Somewhere the no-sys-dirs.patch got disabled, so gcc was looking in
/usr/local/include and /usr/lib. Since I can't fix the patch easily,
I've borrowed the --sysroot trick from clang-wrapper. This causes
builtin paths to be prefixed with /var/empty
(e.g. /var/empty/usr/lib), which don't exist.
This updates the new stable kernel to 3.14, and the new testing kernel
to 3.15.
This also removes the vserver kernel, since it's probably not nearly as
used.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
If the user explictly gives a ref such as "refs/heads/master", `git
rev-parse` failed because we only checked out the `fetchgit`
branch. Now, we also try `git rev-parse fetchgit` if the first call
fails, which fixes the issue.
HipChat (or rather its copy of Qt) expects to find keyboard data in
/usr/share/X11/xkb. So use a LD_PRELOAD library to intercept and
rewrite the Glibc calls that access those paths. We've been doing the
same thing with packages like Spotify, but now this functionality has
been abstracted into a reusable library, libredirect.so. It uses an
environment variable $NIX_REDIRECTS containing a colon-separated list
of path prefixes to be rewritten, e.g. "/foo=bar:/xyzzy=/fnord".
This now provides a handful of different grsecurity kernels for slightly
different 'flavors' of packages. This doesn't change the grsecurity
module to use them just yet, however.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
fetchpatch is fetchurl that determinizes the patch.
Some parts of generated patches change from time to time, e.g. see #1983 and
http://comments.gmane.org/gmane.linux.distributions.nixos/12815
Using fetchpatch should prevent the hash from changing.
Conflicts (auto-solved):
pkgs/development/libraries/haskell/gitit/default.nix
1) Packages formerly called haskell-haskell-platform-ghcXYZ-VVVV.X.Y.Z are
now called haskell-platform-VVVV.X.Y.Z. The latest version can be
installed by running "nix-env -i haskell-platform".
2) The attributes haskellPackages_ghcXYZ.haskellPlatform no longer exist.
Instead, we have attributes like haskellPlatformPackages."2012_4_0_0".
(The last numeric bit must be quoted when used in a Nix file, but not on
the command line to nix-env, nix-build, etc.) The latest Platform has a
top-level alias called simply haskellPlatform.
3) The haskellPackages_ghcXYZ package sets offer the latest version of every
library that GHC x.y.z can compile. For example, if 2.7 is the latest
version of QuickCheck and if GHC 7.0.4 can compile that version, then
haskellPackages_ghc704.QuickCheck refers to version 2.7.
4) All intermediate GHC releases were dropped from all-packages.nix to
simplify our configuration. What remains is a haskellPackages_ghcXYZ set
for the latest version of every major release branch, i.e. GHC 6.10.4,
6.12.3, 7.0.4, 7.2.2, 7.4.2, 7.6.3, 7.8.2, and 7.9.x (HEAD snapshot).
5) The ghcXYZPrefs functions in haskell-defaults.nix now inherit overrides
from newer to older compilers, i.e. an override configured for GHC 7.0.4
will automatically apply to GHC 6.12.3 and 6.10.4, too. This change has
reduced the redundancy in those configuration functions. The downside is
that overriding an attribute for only one particular GHC version has become
more difficult. In practice, this case doesn't occur much, though.
6) The 'cabal' builder has a brand-new argument called 'extension'. That
function is "self : super : {}" by default and users can override it to
mess with the attribute set passed to cabal.mkDerivation. An example use
would be the definition of darcs in all-packages.nix:
| darcs = haskellPackages.darcs.override {
| cabal = haskellPackages.cabal.override {
| extension = self : super : {
| isLibrary = false;
| configureFlags = "-f-library " + super.configureFlags or "";
| };
| };
| };
In this case, extension disables building the library part of the package
to give us an executable-only version that has no dependencies on GHC or
any other Haskell packages.
The 'self' argument refers to the final version of the attribute set and
'super' refers to the original attribute set.
Note that ...
- Haskell Platform packages always provide the Haddock binary that came with
the compiler.
- Haskell Platform 2009.2.0.2 is broken because of build failures in cgi and
cabal-install.
- Haskell Platform 2010.1.0.0 is broken becasue of build failures in cgi.
This function downloads and unpacks a file in one fixed-output
derivation. This is primarily useful for dynamically generated zip
files, such as GitHub's /archive URLs, where the unpacked content of
the zip file doesn't change, but the zip file itself may (e.g. due to
minor changes in the compression algorithm, or changes in timestamps).
Fetchzip is implemented by extending fetchurl with a "postFetch" hook
that is executed after the file has been downloaded. This hook can
thus perform arbitrary checks or transformations on the downloaded
file.
This allows fonts to be installed from anywhere in an unzipped file
rather than having to cd deep into the directory and come back out in
order for e.g. `forceCopy` to work correctly.
This ensures that the intermediate machine is shut down only after the
migration has finished writing the memory dump to disk, to ensure we
don't end up with empty state files depending on how fast the migration
finished before we actually shut down the VM.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This ensures that the builder isn't waiting forever if the Windows VM
drops dead while we're waiting for the controller VM to signal that a
particular command has been executed on the Windows VM. It won't ever
happen in such cases so it doesn't make sense to wait for the timeout.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This hook allows to scatter files in $out to multiple outputs.
For "bin" and "doc" outputs there are prefefined default masks, but
they can be overriden by setting files_<outname>, for example:
files_bin = [ "/bin/*" "/lib/libexec/" ];
To make an effect hook must be specified in buildInputs.
[Note from Austin: I think @edolstra forgot to merge this to master.]
(cherry picked from commit 02b056c5b180b4b8ba22ddc3061d78258e2ef98f on
release-14.04)
These two expressions greatly simplify using the clang-analyzer or
Coverity static analyzer on your C/C++ projects. In fact, they are
identical to nixBuild in every way out of the box, and should 'Just
Work' providing your code can be compiled with Clang already.
The trick is that when running 'make', we actually just alias it to the
appropriate scan build tool, and add a post-build hook that will bundle
up the results appropriately and unalias it.
For Clang, we put the results in $out/analysis and add an 'analysis'
report to $out/nix-support/hydra-build-products pointing to the result
HTML - this means that if the analyzer finds any bugs, the HTML results
will automatically show up Hydra for easy viewing.
For Coverity, it's slightly different. Instead we run the build tool and
after we're done, we tar up the results in a format that Coverity Scan's
service understands. We put the tarball in $out/tarballs under the name
'foo-cov-int.xz' and add an entry for the file to hydra-build-products
as well for easy viewing.
Of course for Coverity you must then upload the build. A Hydra plugin to
do this is on the way, and it will automatically pick up the
cov-int.tar.xz for uploading.
Note that coverityAnalysis requires allowUnfree = true;, as well as the
cov-build tools, which you can download from https://scan.coverity.com -
they're not linked to your account or anything, it's just an annoying
registration wall.
Note this is a first draft. In particular, scan-build fixes the C/C++
compiler to be Clang, and it's perfectly reasonable to want to use Clang
for the analyzer but have scan-build invoke GCC instead.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
This reverts commit a2a398fbda. The
issue *does* still exist in GHC 7.8.2. Compiled binaries have no -rpath
into their own install directory ("$out") and thus cannot find their own
shared libraries. To work around this issue, we pass an explicit -rpath
argument at configure time. We do that only on Linux, though, because
-rpath is known to cause trouble on Darwin, which was the reason I
originally reverted that patch.
This includes a lot of fixes for cross-building to Windows and Mac OS X
and could possibly fix things even for non-cross-builds, like for
example OpenSSL on Windows.
The main reason for merging this in 14.04 already is that we already
have runInWindowsVM in master and it doesn't work until we actually
cross-build Cygwin's setup binary as the upstream version is a fast
moving target which gets _overwritten_ on every new release.
Conflicts:
pkgs/top-level/all-packages.nix
See the comments at f67015cae4
for more information.
Please note: this makes initrd unrepeatable again, but most people will prefer that above an unbootable system.
