Commit Graph

400 Commits

Author SHA1 Message Date
Anders Kaseorg
59b6664f15 Revert "Merge pull request #96254 from Mic92/logging"
This reverts commit 4fc708567f, reversing
changes made to 0e54f3a6d8.

Fixes #96699.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30 15:46:39 -07:00
Anders Kaseorg
a0a421bf5e Revert "Merge pull request #96152 from JJJollyjim/colour-test-machines-staging"
This reverts commit 1bff6fe17c, reversing
changes made to 2995fa48cb.

There’s presumably nothing wrong with this PR, except that it
conflicts with reverting #96254 which broke several tests (#96699).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30 15:45:44 -07:00
aszlig
b0ac24ae41
nixos/test-driver: Use guest time when using sleep
With the Perl driver, machine.sleep(N) was doing a sleep on the guest
machine instead of the host machine. The new Python test driver however
uses time.sleep(), which instead sleeps on the host.

While this shouldn't make a difference most of the time, it *does*
however make a huge difference if the test machine is loaded and you're
sleeping for a minimum duration of eg. an animation.

I stumbled on this while porting most of all my tests to the new Python
test driver and particularily my video game tests failed on a fairly
loaded machine, whereas they don't with the Perl test driver.

Switching the sleep() method to sleep on the guest instead of the host
fixes this.

Signed-off-by: aszlig <aszlig@nix.build>
2020-08-29 00:55:01 +02:00
Florian Klink
0620184f3f nixos/lib/test*: remove perl test driver
This has been deprecated in 20.03, and all tests have been migrated to
the python framework, effectively making this dead code.
2020-08-27 19:45:38 +02:00
Jamie McClymont
d7875caf76 nixos/test: colour machine names 2020-08-27 23:29:55 +12:00
Jörg Thalheim
4fc708567f
Merge pull request #96254 from Mic92/logging 2020-08-26 19:45:24 +01:00
Matthew Bauer
ca3fa9c32a
Merge pull request #95956 from matthewbauer/qemu-cpu-max
runInLinuxVM, test-driver: use -cpu max instead of -cpu host
2020-08-26 12:59:57 -05:00
Jörg Thalheim
87214dbd10
nixos/test-driver: re-introduce log()
Appearantly this is used in tests
2020-08-25 14:50:47 +01:00
Jörg Thalheim
f3c0a09c76
nixos/testdriver: sort imports 2020-08-25 10:15:24 +01:00
Jörg Thalheim
392415c285
nixos/test-driver: switch to pythons' logging lib
- Less code
- more thread-safe according to @flokli
2020-08-25 10:13:27 +01:00
Jörg Thalheim
c1667f85bb
nixos/test-driver: introduce main method
This way we not accidentally use introduce/use global variables.
Also it explictly mark the code for the mypy type checker.
2020-08-25 09:36:37 +01:00
Silvan Mosberger
b77d8ead28
Merge pull request #51850 from roberth/nixos-pure
nixos/lib/eval-config.nix: Add extraModules parameter for opt-in purity
2020-08-24 04:29:37 +02:00
Matthew Bauer
47b56e7c19 runInLinuxVM, test-driver: use -cpu max instead of -cpu host
This appears to avoid requiring KVM when it’s not available. This is
what I originally though -cpu host did. Unfortunately not much
documentation available from the QEMU side on this, but this appears
to square with help:

$ qemu-system-x86 -cpu help
...
x86 host                  KVM processor with all supported host features
x86 max                   Enables all features supported by the accelerator in the current host
...

Whether we actually want to support this not clear, since this only
happens when your CPU doesn’t have full KVM support. Some Nix builders
are lying about kvm support though. Things aren’t too slow without it
though.

Fixes https://github.com/NixOS/nixpkgs/issues/85394

Alternative to https://github.com/NixOS/nixpkgs/pull/83920
2020-08-21 23:42:07 -05:00
Janne Heß
ff03800d3b
nixos/testing: Fix fail() function
The docs say this behaves as succeed(), but it does not return stdout as
succeed() does. This fixes that behaviour
2020-08-21 21:28:24 +02:00
Jacek Galowicz
2a288cb1da
Merge pull request #93824 from blitz/fix-rpi4-installer
Fix Raspberry Pi 4B SD-Card Install Image
2020-08-08 13:45:02 +02:00
Maximilian Bosch
2ae62ef72f
Merge pull request #94611 from Ma27/rel-pkgs-error-msg
nixos/manual: improve error message for invalid values in `relatedPackages'
2020-08-03 20:17:46 +02:00
Maximilian Bosch
6d19c04416
nixos/manual: improve error message for invalid values in `relatedPackages'
As reported in NixOS discourse[1], tracking down invalid values in
`relatedPackages'[2] (i.e. list-items that don't exist in `pkgs`) is
fairly hard as the message "Invalid package attribute path `foobar'"
is quite unhelpful and the trace doesn't point to the source of the
problem either.

This patch improves the error message by mentioning that the issue is an
invalid `relatedPackages`-declaration in $optionName.

[1] https://discourse.nixos.org/t/invalid-package-attribute-path-nextcloud19/8403/9
[2] https://nixos.org/nixpkgs/manual/#sec-functions-library-options
2020-08-03 16:37:59 +02:00
zowoq
25d7880f17 nixos/lib/*: editorconfig fixes 2020-08-04 00:23:54 +10:00
Emery Hemingway
3f922834b8 nixos/lib/make-disk-image.nix: abritary format input
Pass unrecognized format types as the output file extension to
qemu-img. The motivation is support for "vdi" output.
2020-07-31 15:33:08 +02:00
Keshav Kini
5e86bba082 nixos/boot: some documentation improvements
- Give a more accurate description of how fileSystems.<name/>.neededForBoot
  works

- Give a more detailed description of how fileSystems.<name/>.encrypted.keyFile
  works
2020-07-29 14:39:21 -07:00
Lassulus
77cf1a6581
Merge pull request #89331 from Lassulus/make-disk-image
make-disk-image: add hybrid and dynamic sized images
2020-07-28 20:34:20 +02:00
lassulus
883a6079fd make-disk-image: add hybrid and dynamic sized images 2020-07-27 19:45:55 +02:00
Julian Stecklina
fd2047c642 nixos/lib/make-ext4-fs: fix after mkfs.ext4 refactoring
In 9ac1ab10c9 this library function was
refactored to use mkfs.ext4 instead of cptofs. There are two problems:

If populateImageCommands would create no files (which is fine), a cp
invocation would fail due to missing source arguments.

Another problem is that mkfs.ext4 relies on fakeroot to have sane
uid/gids in the generated filesystem image. This currently doesn't
work for cross compiling.
2020-07-26 22:49:55 +02:00
Timo Kaufmann
830a8d6ee1
Merge pull request #74174 from raboof/fix-74047-stable-gpt-disk-guid
make-iso9660-image: produce stable GPT disk GUID
2020-07-21 00:30:57 +02:00
Bas van Dijk
6e7822b8f3 lib: toHex -> toHexString & toBase -> toBaseDigits
This makes the type of these functions more apparent from the name.
2020-07-20 13:14:19 +02:00
Bas van Dijk
e15815e885 nixos/tests/networking.nix: test the services.dhcpd4.machines option
This modifies the `router` to not give out a range of IP addresses but
only give out a fixed address based on the MAC address using the
`services.dhcpd4.machines` option.

To get access to the MAC address the `qemuNicMac` function is defined
and exported from `qemu-flags.nix`.
2020-07-20 13:09:27 +02:00
Bas van Dijk
0410f5dff9 nixos/tests: support up to 255 nodes in NixOS tests 2020-07-20 13:09:27 +02:00
Arnout Engelen
be006eab1f
make-iso9660-image: produce stable GPT disk GUID
By generating a version-5 GUID based on $out (which contains
the derivation hash) and preventing isohybrid from overwriting
the GPT table (which already is populated correctly by xorriso).

Tested by:
* booting from USB disk on a UEFI system
* booting from USB disk on a non-UEFI system
* booting from CD on a UEFI system
* booting from CD on a non-UEFI system
* booting from CD on an OSX system

Also tested that "nix-build ./nixos/release-combined.nix -A
nixos.iso_minimal.x86_64-linux -I nixpkgs=~/nixpkgs-r13y --check"
now succeeds.

Fixes #74047
2020-07-20 11:16:59 +02:00
lassulus
8d05772134
nixos make-disk-image: fix permissions of /build
This was broken in 460c0d6 (PR #90431); now the nixos-unstable channel
should get unblocked.
vcunat modified this commit to use env-var instead of hardcoding /build
2020-07-10 10:31:26 +02:00
Justinas Stankevičius
f815cb6adf
nixos/test-driver: print a traceback when testScript fails (#92369)
* print a traceback: assertion message can be empty

* change print back to eprint
2020-07-06 23:29:40 +02:00
Samuel Dionne-Riel
736c7ca712
Merge pull request #82718 from misuzu/armv7l-ext4-fs-fix
nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs
2020-07-01 21:38:07 -04:00
misuzu
9ac1ab10c9 nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs
This fixes image creation on armv7l when image is bigger than 2G.
Also fix some reproducibility issues and other cptofs issues.
2020-07-01 11:32:28 +03:00
Michele Guerini Rocco
dab676b2d7
Merge pull request #65231 from buckley310/grub-password
grub: add support for passwords
2020-07-01 09:04:30 +02:00
Alexandre Esteves
e10e7d6a8b
testing-python: fix typo 2020-06-30 22:31:32 -05:00
Florian Klink
9e248c9ec9
Merge pull request #91046 from NinjaTrappeur/nin-delete-vm-state
test-driver.py: delete VM state directory after test run
2020-06-28 18:41:38 +02:00
rnhmjoj
b520055df6
nixos/lib/test-driver: add wait_for_console_text
This method is similar to wait_for_text but is based on matching
serial console lines instead of the VGA output.
2020-06-24 10:22:53 +02:00
Félix Baylac-Jacqué
7e7aa529d9
test-driver.py: delete VM state directory after test run
Keeping the VM state test across several run sometimes lead to subtle
and hard to spot errors in practice. We delete the VM state which
contains (among other things) the qcow volume.

We also introduce a -K (--keep-vm-state) flag making VM state to
persist after the test run. This flag makes test-driver.py to match
its previous behaviour.
2020-06-21 10:27:51 +02:00
Florian Klink
079de14fd3 nixos/make-ext4-fs: increase fudge factor from 1.03 to 1.10
Turns out, on smaller images (~800MiB uncompressed sdcard image size),
the current fudge factor is way too small to even get the system to the
phase where it can resize itself.

I first tried with 1.05, but it wasn't enough.
2020-06-21 09:47:02 +02:00
Peter Hoeg
eb62c7edc8
Merge pull request #90261 from prusnak/installer-zstd-rm
images: remove original files when using zstd for compression
2020-06-14 10:39:10 +08:00
Pavol Rusnak
490fc040be
images: remove original files when using zstd for compression 2020-06-14 00:19:11 +02:00
Emery Hemingway
b8a9e3eea1 nixos/lib/make-disk-image.nix: VDI support 2020-06-13 11:08:48 +05:30
Finn Behrens
f73436b374
nixos/make-iso9660: fix zstd input 2020-06-09 11:39:26 +02:00
Marek Mahut
7b9d7cc05d
Merge pull request #85947 from prusnak/images-zstd
Use zstd for ISO and SD images
2020-06-07 19:09:43 +02:00
Florian Klink
8a388c8296
Merge pull request #82258 from erikarvstedt/fix-xchg-caching
fix inconsistent caching of VM xchg dirs
2020-06-01 22:34:48 +02:00
Erik Arvstedt
d85dc4f690
qemu-vm: fix inconsistent caching of xchg dirs
xchg is advertised as a bidirectional exchange dir, but file content
transfer from host to VM fails due to caching:
If a file is read in the VM and then modified on the host, subsequent
re-reads in the VM can yield old, cached data.
This is caused by the use of 9p's cache=loose mode that is explicitly
meant for read-only mounts.

9p doesn't provide any suitable cache modes, so fix this by disabling
caching.

Also, remove a now unnecessary sync in the test driver.
2020-06-01 21:55:33 +02:00
Erik Arvstedt
dfa743e49e
test-driver: remove useless syncs
These syncs have the goal to transfer host filesystem changes to the VM,
but they have no effect because 1) syncing in the VM can't possibly pull
in host data and 2) 9p is accessing the host filesystem on the cached
layer anyways, so even syncing on the host would have no effect in the
VM.
2020-05-27 23:03:15 +02:00
Eelco Dolstra
0ffc85d64b
Remove unused files 2020-05-14 14:34:50 +02:00
Chuck
751a27020e nixos/test-driver: Specify /bin/sh shell when running a bourne shell script as the user
The test harness provides the commands it wishes to run in Bourne
syntax.  This fails if the user uses a different shell.  For example,
with fish:

  machine.wait_for_unit("graphical-session.target", "alice")

machine # fish: Unsupported use of '='. To run '-u`' with a modified environment, please use 'env XDG_RUNTIME_DIR=/run/user/`id -u`…'
machine # XDG_RUNTIME_DIR=/run/user/`id -u` systemctl --user --no-pager show "graphical-session.target"
machine # ^
machine # [   16.329957] su[1077]: pam_unix(su:session): session closed for user alice
error: retrieving systemctl info for unit "graphical-session.target" under user "alice" failed with exit code 127
2020-05-09 11:01:17 -07:00
Eelco Dolstra
3f80fadec4 testing{-python}.nix: Remove log pretty-printing cruft
This completes the removal of the nested log feature, which previously
got removed from Nix, Hydra, stdenv and GNU Make. In particular, this
means that the output of VM builds no longer contains a copy of
jQuery.
2020-05-07 15:56:30 +02:00
Eelco Dolstra
78f2a83029 test-driver.py: Fix deadlock when the log queue gets full
If a program (e.g. nixos-install) writes more than 1000 lines to
stderr during execute(), then process_serial_output() deadlocks
waiting for the queue to be processed. So use an unbounded queue
instead.

We should probably get rid of the structured log output (log.xml),
since then we don't need the log queue anymore.
2020-05-07 15:25:24 +02:00