Commit Graph

368 Commits

Author SHA1 Message Date
sternenseemann
e2372502d3 netcat: make netcat-openbsd the default netcat (#19411)
The motivation for this change is the following: As gnu-netcat,
e. g. does not support ipv6, it is not suitable as default netcat.

This commit also fixes all obvious build issues caused by this change.
2016-10-30 15:06:04 +01:00
Vladimír Čunát
027efec879 Merge staging without python splitting for now
The split needs more time to finish rebuilding,
but the rest seems OK and there are security fixes.
2016-10-14 09:24:21 +02:00
Profpatsch
bef6bef0d2
stdenv/stripHash: print to stdout, not to variable
`stripHash` documentation states that it prints out the stripped name to
the stdout, but the function stored the value in `strippedName`
instead.

Basically all usages did something like
`$(stripHash $foo | echo $strippedName)` which is just braindamaged.
Fixed the implementation and all invocations.
2016-10-11 18:34:36 +02:00
Allen Nelson
4abe579250 add docs to docker build functions
bring back ls_tar

replace goPackages with go

don't hardcode /nix/store in vmTools

more docs
2016-09-29 12:52:57 -05:00
Eelco Dolstra
f081a1aaf4 debian: 8.5 -> 8.6 2016-09-22 15:53:29 +02:00
Nikolay Amiantov
698cadd714 runVM: mount devpts 2016-09-04 17:11:01 +03:00
Nikolay Amiantov
8b38b6aae2 runVM: check exit code before postVM eval 2016-09-04 17:11:01 +03:00
Tuomas Tynkkynen
74a3a2cd7e treewide: Use makeBinPath 2016-08-23 01:18:10 +03:00
Domen Kožar
b9e009b5b5 add Fedora 24 2016-08-12 16:44:23 +02:00
Maarten Hoogendoorn
e809667b17 vmTools.runInLinuxImage: add virtio_rng device
This allows the QEMU VM's to use the /dev/random device, by
getting entropy from the host.
2016-07-31 12:39:19 +02:00
Tuomas Tynkkynen
13160d9c10 buildInLinuxVM: Workaround bug #16742
This is blocking the channel update: #16949
2016-07-14 15:12:50 +03:00
zimbatm
4f5918cd2e Revert "stdenv: introduce baseHash() to replace stripHash()"
Introduced by mistake

This reverts commit e71a5cb878.
2016-06-25 14:25:58 +01:00
zimbatm
e71a5cb878 stdenv: introduce baseHash() to replace stripHash()
stripHash uses a global variable to communicate it's computation
results, but it's not necessary. You can just pipe to stdout in a
subshell. A function mostly behaves like just another command.

baseHash() also introduces a suffix-stripping capability since it's
something the users of the function tend to use.
2016-06-25 14:20:56 +01:00
Eelco Dolstra
5fc64c4baa debian: 7.10 -> 7.11, 8.4 -> 8.5 2016-06-20 14:39:44 +02:00
zimbatm
b0f8416c5c Merge pull request #16180 from zimbatm/shell-escaping
Escape all shell arguments uniformly
2016-06-19 23:27:52 +01:00
Rob Vermaas
91436641ec Fix hash for Debian 8.4 Jessie
(cherry picked from commit fd60751ce0c85427423b78d8a46c3f78d65bd0e2)
2016-06-13 12:20:55 +00:00
zimbatm
28fa4a2f03 Escape all shell arguments uniformly 2016-06-12 18:11:37 +01:00
Domen Kožar
56714859f4 add CentOS 7.1 2016-05-24 11:35:39 +01:00
Domen Kožar
7fc845aeb1 add OpenSuse 13.2
(cherry picked from commit 2cf5dcd99a7d3aac8a39ab98c1738454dfa20bfb)
Signed-off-by: Domen Kožar <domen@dev.si>
2016-05-24 11:06:11 +01:00
Domen Kožar
ba0d4ecaf7 debian7: change hash due to 7.10 release
(cherry picked from commit 00df301ac2fd1818fa1f96debcee23dbb979834d)
Signed-off-by: Domen Kožar <domen@dev.si>
2016-05-24 10:40:39 +01:00
Carles Pagès
e7ab828da1 makeImageFromDebDist: accept additional parameters for vm, as in rpm version. 2016-05-11 15:43:24 +02:00
Domen Kožar
8a3b70791c vmTools.diskImages: add ubuntu 16.04 2016-04-29 11:50:27 +01:00
Tuomas Tynkkynen
01854a850a treewide: Replace module_init_tools -> kmod
The former is deprecated and doesn't handle compressed kernel modules,
so all current usages of it are broken.
2016-04-22 10:40:57 +03:00
Vladimír Čunát
30f14243c3 Merge branch 'master' into closure-size
Comparison to master evaluations on Hydra:
  - 1255515 for nixos
  - 1255502 for nixpkgs
2016-04-10 11:17:52 +02:00
Vladimír Čunát
d1df28f8e5 Merge 'staging' into closure-size
This is mainly to get the update of bootstrap tools.
Otherwise there were mysterious segfaults:
https://github.com/NixOS/nixpkgs/pull/7701#issuecomment-203389817
2016-04-07 14:40:51 +02:00
Vladimír Čunát
aa670eb503 vmTools: update debian jessie 8.3 -> 8.4
Their in-place updates break download hashes...
2016-04-05 14:32:04 +02:00
Lluís Batlle i Rossell
635c99ce87 vm: allow overriding QEMU_OPTS / memSize for images.
It's nice to be able to create disk images with -smp 4
in qemu.
2016-04-01 10:32:59 +02:00
Vladimír Čunát
ab15a62c68 Merge branch 'master' into closure-size
Beware that stdenv doesn't build. It seems something more will be needed
than just resolution of merge conflicts.
2016-04-01 10:06:01 +02:00
Lluís Batlle i Rossell
ab93f8c137 Making vm's qemu cache=unsafe. Faster.
I don't think it's unsafe, if it's meant for nix expressions.
2016-03-31 09:27:25 +02:00
Lluís Batlle i Rossell
e21dd19168 Making vm's interactive shell handle the terminal well. 2016-03-31 09:27:14 +02:00
Nicolas B. Pierron
6313a5698a Replace references to all-packages.nix, by references to the top-level of nixpkgs repository. 2016-03-13 18:25:52 +00:00
Vladimír Čunát
09af15654f Merge master into closure-size
The kde-5 stuff still didn't merge well.
I hand-fixed what I saw, but there may be more problems.
2016-03-08 09:58:19 +01:00
Lluís Batlle i Rossell
202ebf794c vm/rpm/rpm-closure.pl: make it deterministic
Some recent perl version introduced "keys" to return the keys
in random order. As some of the packages are solved by "provides" and
based on the order, this randomness affects what packages get into the
closure.

This problem may be in other nix perl scripts.
2016-03-01 11:02:42 +01:00
Cole Mickens
718848d5aa azure: package qemu @ 2.2.0
This commit packages qemu-220. This package is qemu-2.2.0
and is only used with Azure.
2016-02-18 21:08:28 -08:00
Vladimír Čunát
d039c87984 Merge branch 'master' into closure-size 2016-02-14 08:33:51 +01:00
Tuomas Tynkkynen
3cc7fa0be9 treewide: Mass replace 'lzma}/bin' to refer the 'bin' output 2016-02-01 20:46:15 +02:00
Eelco Dolstra
9855cbcbf8 debian: 8.2 -> 8.3 2016-01-30 11:08:37 +01:00
Vladimír Čunát
f9f6f41bff Merge branch 'master' into closure-size
TODO: there was more significant refactoring of qtbase and plasma 5.5
on master, and I'm deferring pointing to correct outputs to later.
2015-12-31 09:53:02 +01:00
aszlig
a5bc11f9eb
nixos/vm-tests: Remove msize mount option
This seems to be the root cause of the random page allocation failures
and @wizeman did a very good job on not only finding the root problem
but also giving a detailed explanation of it in #10828.

Here is an excerpt:

  The problem here is that the kernel is trying to allocate a contiguous
  section of 2^7=128 pages, which is 512 KB. This is way too much:
  kernel pages tend to get fragmented over time and kernel developers
  often go to great lengths to try allocating at most only 1 contiguous
  page at a time whenever they can.

  From the error message, it looks like the culprit is unionfs, but this
  is misleading: unionfs is the name of the userspace process that was
  running when the system ran out of memory, but it wasn't unionfs who
  was allocating the memory: it was the kernel; specifically it was the
  v9fs_dir_readdir_dotl() function, which is the code for handling the
  readdir() function in the 9p filesystem (the filesystem that is used
  to share a directory structure between a qemu host and its VM).

  If you look at the code, here's what it's doing at the moment it tries
  to allocate memory:

    buflen = fid->clnt->msize - P9_IOHDRSZ;

    rdir = v9fs_alloc_rdir_buf(file, buflen);

  If you look into v9fs_alloc_rdir_buf(), you will see that it will try
  to allocate a contiguous buffer of memory (using kzalloc(), which is a
  wrapper around kmalloc()) of size buflen + 8 bytes or so.

  So in reality, this code actually allocates a buffer of size
  proportional to fid->clnt->msize. What is this msize? If you follow
  the definition of the structures, you will see that it's the
  negotiated buffer transfer size between 9p client and 9p server. On
  the client side, it can be controlled with the msize mount option.

  What this all means is that, the reason for running out of memory is
  that the code (which we can't easily change) tries to allocate a
  contiguous buffer of size more or less equal to "negotiated 9p
  protocol buffer size", which seems to be way too big (in our NixOS
  tests, at least).

After that initial finding, @lethalman tested the gnome3 gdm test
without setting the msize parameter at all and it seems to have resolved
the problem.

The reason why I'm committing this without testing against all of the
NixOS VM test is basically that I think we can only go better but not
worse than the current state.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-12-14 17:26:24 +01:00
Luca Bruno
a412927924 Merge remote-tracking branch 'origin/master' into closure-size 2015-11-25 21:37:30 +01:00
Eelco Dolstra
75e41b0210 Add Fedora 23 2015-11-25 16:18:15 +01:00
Eelco Dolstra
e4eee41ad0 Add Ubuntu 15.10 2015-11-25 15:40:08 +01:00
Vladimír Čunát
333d69a5f0 Merge staging into closure-size
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
2015-11-20 14:32:58 +01:00
aszlig
82d88b5bcb
vmTools.commonDebPackages: Add "mawk".
While debugging an issue with running NixOps tests, I found out that the
output from debClosureGenerator is not deterministic.

The reason behind this is the way how Provides and Replaces fields are
handled. I haven't yet found out what's the exact issue, but so far
packages "Provides" are more or less picked at random.

So, running the NixOps Hetzner tests we get either mawk, original-awk or
gawk altering on every invocation.

While for the test it isn't poisionous whether wi have mawk or gawk,
having original-awk certainly is, because live-build only works with
mawk or gawk.

The best solution would obviously be to make debClosureGenerator
deterministic, but in the case of "Provides: awk", we can safely pick
mawk by default, because the latter has a "Priority: required" in its
package description.

This also has the advantage that we can safely cherry-pick this to
release-15.09 because it's very unlikely that we'll break the
debClosureGenerator by adding a dependency to commonDebPackages.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-10-26 20:56:42 +01:00
Eelco Dolstra
7260c04266 debian: Update to 7.9, 8.2 2015-10-08 11:20:26 +02:00
Eelco Dolstra
106738b196 Give more memory for the disk image builder
http://hydra.nixos.org/build/26480662
2015-10-06 15:03:26 +02:00
Vladimír Čunát
5227fb1dd5 Merge commit staging+systemd into closure-size
Many non-conflict problems weren't (fully) resolved in this commit yet.
2015-10-03 13:33:37 +02:00
Eelco Dolstra
69829dd4ff Debian: Update to 8.1 2015-06-12 11:22:59 +02:00
Eelco Dolstra
99d2e80d97 In VM builds, connect stdin to /dev/null
This prevents programs from hanging while trying to read from stdin.
2015-06-04 20:08:56 +02:00
Benjamin Staffin
1db5bafc94 Typo fix! s/wheezy/jessie/ 2015-05-29 11:16:33 +02:00
Benjamin Staffin
c7ac0ed3eb Add Debian 8.0
Had to add file type logic in debClosureGenerator because Packages.bz2
is not available for Jessie - only Packages.xz is.
2015-05-29 10:49:20 +02:00
Eelco Dolstra
178c52e7ad Add Ubuntu 15.04 2015-05-22 13:21:02 +02:00
Eelco Dolstra
fef86de618 Make module loading work in VMs
Also, execute stage 2 as PID 1.
2015-05-13 17:36:58 +02:00
Eelco Dolstra
b97362989f Provide /run in VMs 2015-05-13 17:36:29 +02:00
Eelco Dolstra
c3f6c8c582 Mount /dev/shm in VMs 2015-05-07 15:31:36 +02:00
Vladimír Čunát
3b9ef2c71b fix "libc}/lib" and similar references
Done mostly without any verification.
I didn't bother with libc}/include, as the path is still correct.
2015-05-05 11:52:08 +02:00
Eelco Dolstra
3a31043b43 fillDiskWithRPMs: Pass --extract-over-symlinks to cpio
http://hydra.nixos.org/build/20003320
2015-02-27 14:00:11 +01:00
Eelco Dolstra
fa47d347a4 Fix conflict between fedora-release and generic-release
http://hydra.nixos.org/build/19594340
2015-02-12 17:55:29 +01:00
Eelco Dolstra
ff37dc6fb6 Remove tabs 2015-02-12 17:55:29 +01:00
Eelco Dolstra
28ea9050fa Remove ancient RedHat 9 / SuSE 9 images
These are 12 years old and I don't think anybody is using them.
2015-02-12 17:55:29 +01:00
Eelco Dolstra
da73688e3d fillDiskWithRPMs: Shut up some warnings 2015-02-12 17:55:28 +01:00
Eelco Dolstra
3786f08d25 Add Fedora 21 2015-02-10 11:32:43 +01:00
Eelco Dolstra
1b1e015e6e Update Debian to 7.8 2015-01-15 14:51:55 +01:00
Eelco Dolstra
7358b8dfda Fix a warning in initrd generation
${pkgs.busybox}/bin and ${pkgs.busybox}/sbin are now the same
directory, so don't copy both.
2015-01-12 17:52:30 +01:00
Eelco Dolstra
872f8c0431 Remove RHEL 7 RC
It no longer exists on the RedHat FTP site.
2015-01-12 15:13:21 +01:00
Eelco Dolstra
02df7cdaa0 Fix CentOS 6.5 URLs 2015-01-12 15:13:05 +01:00
Eelco Dolstra
b8e33a5dcb Ubuntu 14.10: Update hashes 2014-12-05 16:02:59 +01:00
Eelco Dolstra
99a2afd784 Update to Debian 7.7 2014-11-20 11:11:21 +01:00
Eelco Dolstra
e5513ac4dc Add Ubuntu 14.10 2014-11-20 11:11:08 +01:00
Eelco Dolstra
fda5c17a6f vmTools: Fix permissions on /tmp and /dev/* 2014-11-18 14:49:39 +01:00
Shea Levy
3fe41ca763 Debain 7.6-amd64 tarball was updated in place 2014-10-20 15:26:51 -04:00
Shea Levy
652030e85d debian packages tarball was updated in place 2014-10-20 15:23:44 -04:00
Eelco Dolstra
91fa55e702 Remove references to <nixpkgs>
Nixpkgs should not use <nixpkgs> to refer to itself, because $NIX_PATH
might not be set or not point to the same Nixpkgs. It's also
unnecessary.
2014-09-23 15:48:15 +02:00
Mathijs Kwik
e80cbedbae VM builds: Use default kernel
(this was set to 3.10 when 3.4 was our default)
See commit d846e97656.
2014-09-23 09:17:01 +02:00
Eelco Dolstra
7e744041f3 rpm-closure.pl: Reduce verbosity 2014-09-05 12:37:07 +02:00
Eelco Dolstra
05e81e0d9f debian: Update to 6.0.10 2014-08-10 15:17:00 +02:00
Eelco Dolstra
ff97b7dbe6 Merge remote-tracking branch 'origin/master' into staging 2014-07-14 17:04:55 +02:00
Eelco Dolstra
1245ca3ff7 debian: Update to 7.6 2014-07-14 14:35:09 +02:00
Eelco Dolstra
40f7b0f9df Another attempt to eradicate ensureDir
See c556a6ea46.
2014-06-30 14:56:10 +02:00
Eelco Dolstra
139608dd34 Update Ubuntu Packages.bz2 hashes 2014-05-20 13:39:06 +02:00
aszlig
97dc8a88e5
vm/windows: Wait for migration to finish.
This ensures that the intermediate machine is shut down only after the
migration has finished writing the memory dump to disk, to ensure we
don't end up with empty state files depending on how fast the migration
finished before we actually shut down the VM.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-05-07 07:21:10 +02:00
aszlig
bd78e674c5
vm/windows: Exit if VM has dropped out.
This ensures that the builder isn't waiting forever if the Windows VM
drops dead while we're waiting for the controller VM to signal that a
particular command has been executed on the Windows VM. It won't ever
happen in such cases so it doesn't make sense to wait for the timeout.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-05-07 07:21:10 +02:00
Eelco Dolstra
0ff68e81b9 debian: Update to 7.5
[Note from Austin: I think @edolstra forgot to merge this to master.]

(cherry picked from commit 02b056c5b180b4b8ba22ddc3061d78258e2ef98f on
release-14.04)
2014-05-03 10:55:26 -05:00
Austin Seipp
2be1b4c034 build-support/vm: add Ubuntu 14.04 Trusty Tahir
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2014-05-03 10:51:32 -05:00
Rob Vermaas
eaedf303a0 Add RHEL7 to vm functions. 2014-05-01 15:13:35 +02:00
Eelco Dolstra
1bae93cb82 debian: Update to 6.0.9 2014-04-25 10:43:26 +02:00
Eelco Dolstra
f73ff0529e runInLinuxVM: Set the clock properly 2014-03-18 15:02:09 +01:00
aszlig
bd2d120c08
vm/windows: Allow to specify a different arch.
So far, we determined this based on stdenv.is64bit, but there are cases
where you want to run/build a 32bit program on a 64 bit Windows.

This is now possible, by passing windowsImage.arch = "i686" | "x86_64"
to runInWindowsVM. Based an what was passed, the corresponding Cygwin
packages and setup.exe are bootstrapped.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-04 09:22:18 +01:00
aszlig
e86ffa08b7
vm/windows: Update SHAs for Cygwin setup.ini.
Another very annoying part. Unfortunately, the only option we might have
here is to include it in nixpkgs or maybe make a fixed Hash on the
result of the closure fetcher.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 23:37:41 +01:00
aszlig
5bd52771ba
vm/windows: Replace binary with cross cygwinSetup.
As the official Cygwin setup binary download doesn't come in snapshots
or even versioned, the fetchurl of setup.exe will frequently fail, which
in turn will annoy us as hell (or at least me).

One warning though: The fetchurl is currently broken and the cross-build
might not work yet for example on mingw32 (mingw-w64 branch on its way),
but the upstream URL has already changed and the new version contains a
bug (not yet tracked down) which breaks our Windows bootstrap process.

So to conclude: If it's already broken, make it at least "less broken".

"Not broken" is coming soon with the merge of the mingw-w64 branch.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 23:21:34 +01:00
aszlig
40635ed343
vm/windows: Remove stray "import <nixpkgs>".
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 22:45:14 +01:00
aszlig
ca8cdcda06
vm/windows: Fix typo in runInWindowsVM.
Coincidentally, even with this typo, most tests work anyway, so I didn't
notice it in the first place.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 22:42:10 +01:00
aszlig
88ea17284e
vmTools: Hook in runInWindowsVM function.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 06:21:48 +01:00
aszlig
895a683d39
vm/windows: Add setup.ini for 32bit architecture.
Also update 64bit setup.ini and check whether we have a 64 bit stdenv in
order to choose the proper Cygwin version. Otherwise we now have the
setup.ini for 32bit available as well.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 06:20:24 +01:00
aszlig
623f1940c0
vm/windows: Make list of dependencies explicit.
So far, the VMs have always been using the native architecture, because
it was reimporting <nixpkgs> several times. Now, we propagate a list of
packages down to all sub-imports, which not only makes clearer which
dependencies a part actually has, but also will make it easier in case
we want to refactor those parts to use callPackage.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 05:43:34 +01:00
aszlig
d8e66722a3
vm/windows: Factor out bootstrapping process.
This now isolates the vmTools integration from the bootstrap process and
thus removes our fixed Windows ISO and product key. The latter can now
be provided by an attribute "windowsImage" to runInWindowsVM.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:01 +01:00
aszlig
e40f41e505
vm/windows: Collect exit code from xchg.
This is the last item that was missing to get a fully working
runInWindowsVM function. Apart from checking exit codes, we also now
have preVM/postVM hooks which we can use to write arbitrary constructs
around this architecture, without the need to worry about specific
details.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:00 +01:00
aszlig
b01c9624cf
vm/windows: Add new runInWindowsVM function.
This function is quite similar to runInLinuxVM, but also ensures that
the builder is run decoupled of the Nix store and using the userland
inside the VM.

We're now picking up the environment variables saved in the previous
commit.

The reason we suppress all errors from the source operation is that it
would emit a ton of errors because we're trying to set read-only
variables.

Also, detecting whether the origBuilder is using the default builder
from the stdenv is currently a bit of a workaround until we have a
specialized pseudo-cross-stdenv someday in the future[TM].

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:00 +01:00
aszlig
dd8b0fcf52
vm/windows: Save envirenoment before running VM.
Later, when we start the actual builder, we're going to restore those
environment variables. We're using "(set; declare -p)", here, because
the former is just printing _all_ environment variables, even those not
supported, and the latter only lists specifically declared variables,
which also encludes exports.

The "declare -p" command also emits those variables in a format similar
to the "export" command.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:00 +01:00
aszlig
707b7ad1bd
vm/windows: Generate mounts from an attribute set.
This is mainly to make it easier to quickly change mappings, without
making room for errors such as typos.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:59 +01:00