Commit Graph

423 Commits

Author SHA1 Message Date
Arnout Engelen
be006eab1f
make-iso9660-image: produce stable GPT disk GUID
By generating a version-5 GUID based on $out (which contains
the derivation hash) and preventing isohybrid from overwriting
the GPT table (which already is populated correctly by xorriso).

Tested by:
* booting from USB disk on a UEFI system
* booting from USB disk on a non-UEFI system
* booting from CD on a UEFI system
* booting from CD on a non-UEFI system
* booting from CD on an OSX system

Also tested that "nix-build ./nixos/release-combined.nix -A
nixos.iso_minimal.x86_64-linux -I nixpkgs=~/nixpkgs-r13y --check"
now succeeds.

Fixes #74047
2020-07-20 11:16:59 +02:00
lassulus
8d05772134
nixos make-disk-image: fix permissions of /build
This was broken in 460c0d6 (PR #90431); now the nixos-unstable channel
should get unblocked.
vcunat modified this commit to use env-var instead of hardcoding /build
2020-07-10 10:31:26 +02:00
Justinas Stankevičius
f815cb6adf
nixos/test-driver: print a traceback when testScript fails (#92369)
* print a traceback: assertion message can be empty

* change print back to eprint
2020-07-06 23:29:40 +02:00
Samuel Dionne-Riel
736c7ca712
Merge pull request #82718 from misuzu/armv7l-ext4-fs-fix
nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs
2020-07-01 21:38:07 -04:00
misuzu
9ac1ab10c9 nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs
This fixes image creation on armv7l when image is bigger than 2G.
Also fix some reproducibility issues and other cptofs issues.
2020-07-01 11:32:28 +03:00
Michele Guerini Rocco
dab676b2d7
Merge pull request #65231 from buckley310/grub-password
grub: add support for passwords
2020-07-01 09:04:30 +02:00
Alexandre Esteves
e10e7d6a8b
testing-python: fix typo 2020-06-30 22:31:32 -05:00
Florian Klink
9e248c9ec9
Merge pull request #91046 from NinjaTrappeur/nin-delete-vm-state
test-driver.py: delete VM state directory after test run
2020-06-28 18:41:38 +02:00
rnhmjoj
b520055df6
nixos/lib/test-driver: add wait_for_console_text
This method is similar to wait_for_text but is based on matching
serial console lines instead of the VGA output.
2020-06-24 10:22:53 +02:00
Félix Baylac-Jacqué
7e7aa529d9
test-driver.py: delete VM state directory after test run
Keeping the VM state test across several run sometimes lead to subtle
and hard to spot errors in practice. We delete the VM state which
contains (among other things) the qcow volume.

We also introduce a -K (--keep-vm-state) flag making VM state to
persist after the test run. This flag makes test-driver.py to match
its previous behaviour.
2020-06-21 10:27:51 +02:00
Florian Klink
079de14fd3 nixos/make-ext4-fs: increase fudge factor from 1.03 to 1.10
Turns out, on smaller images (~800MiB uncompressed sdcard image size),
the current fudge factor is way too small to even get the system to the
phase where it can resize itself.

I first tried with 1.05, but it wasn't enough.
2020-06-21 09:47:02 +02:00
Peter Hoeg
eb62c7edc8
Merge pull request #90261 from prusnak/installer-zstd-rm
images: remove original files when using zstd for compression
2020-06-14 10:39:10 +08:00
Pavol Rusnak
490fc040be
images: remove original files when using zstd for compression 2020-06-14 00:19:11 +02:00
Emery Hemingway
b8a9e3eea1 nixos/lib/make-disk-image.nix: VDI support 2020-06-13 11:08:48 +05:30
Finn Behrens
f73436b374
nixos/make-iso9660: fix zstd input 2020-06-09 11:39:26 +02:00
Marek Mahut
7b9d7cc05d
Merge pull request #85947 from prusnak/images-zstd
Use zstd for ISO and SD images
2020-06-07 19:09:43 +02:00
Florian Klink
8a388c8296
Merge pull request #82258 from erikarvstedt/fix-xchg-caching
fix inconsistent caching of VM xchg dirs
2020-06-01 22:34:48 +02:00
Erik Arvstedt
d85dc4f690
qemu-vm: fix inconsistent caching of xchg dirs
xchg is advertised as a bidirectional exchange dir, but file content
transfer from host to VM fails due to caching:
If a file is read in the VM and then modified on the host, subsequent
re-reads in the VM can yield old, cached data.
This is caused by the use of 9p's cache=loose mode that is explicitly
meant for read-only mounts.

9p doesn't provide any suitable cache modes, so fix this by disabling
caching.

Also, remove a now unnecessary sync in the test driver.
2020-06-01 21:55:33 +02:00
Erik Arvstedt
dfa743e49e
test-driver: remove useless syncs
These syncs have the goal to transfer host filesystem changes to the VM,
but they have no effect because 1) syncing in the VM can't possibly pull
in host data and 2) 9p is accessing the host filesystem on the cached
layer anyways, so even syncing on the host would have no effect in the
VM.
2020-05-27 23:03:15 +02:00
Eelco Dolstra
0ffc85d64b
Remove unused files 2020-05-14 14:34:50 +02:00
Chuck
751a27020e nixos/test-driver: Specify /bin/sh shell when running a bourne shell script as the user
The test harness provides the commands it wishes to run in Bourne
syntax.  This fails if the user uses a different shell.  For example,
with fish:

  machine.wait_for_unit("graphical-session.target", "alice")

machine # fish: Unsupported use of '='. To run '-u`' with a modified environment, please use 'env XDG_RUNTIME_DIR=/run/user/`id -u`…'
machine # XDG_RUNTIME_DIR=/run/user/`id -u` systemctl --user --no-pager show "graphical-session.target"
machine # ^
machine # [   16.329957] su[1077]: pam_unix(su:session): session closed for user alice
error: retrieving systemctl info for unit "graphical-session.target" under user "alice" failed with exit code 127
2020-05-09 11:01:17 -07:00
Eelco Dolstra
3f80fadec4 testing{-python}.nix: Remove log pretty-printing cruft
This completes the removal of the nested log feature, which previously
got removed from Nix, Hydra, stdenv and GNU Make. In particular, this
means that the output of VM builds no longer contains a copy of
jQuery.
2020-05-07 15:56:30 +02:00
Eelco Dolstra
78f2a83029 test-driver.py: Fix deadlock when the log queue gets full
If a program (e.g. nixos-install) writes more than 1000 lines to
stderr during execute(), then process_serial_output() deadlocks
waiting for the queue to be processed. So use an unbounded queue
instead.

We should probably get rid of the structured log output (log.xml),
since then we don't need the log queue anymore.
2020-05-07 15:25:24 +02:00
Florian Klink
3cdd558a6f nixosTests: drop nr_tests and failed_tests variables
With the tests now bailing out early on a failing subtest, we don't need
to keep a list of failed tests, or the number of total tests
2020-05-01 01:22:42 +02:00
Jacek Galowicz
b139bc0d87 nixosTests: Reraise exception in subtests 2020-05-01 01:22:42 +02:00
Pavol Rusnak
f20fd89fc0
iso-image: use zstd for compression 2020-04-24 18:34:11 +02:00
Michael Reilly
84cf00f980
treewide: Per RFC45, remove all unquoted URLs 2020-04-10 17:54:53 +01:00
worldofpeace
a775961c8f Revert "Revert "iso-image: normalize volumeID""
This reverts commit 21b50e6f68.
2020-04-09 20:06:32 -04:00
worldofpeace
21b50e6f68
Revert "iso-image: normalize volumeID" 2020-04-09 19:43:59 -04:00
worldofpeace
8583d99e42
Merge pull request #83551 from worldofpeace/iso-id
iso-image: normalize volumeID
2020-04-09 19:31:52 -04:00
Silvan Mosberger
85e866db6f
nixos/lib/test-driver: Fix require_unit_state hardcoded formatting 2020-04-06 23:55:04 +02:00
Jan Tojnar
cab6b019b1
Revert "make-options-doc: fix string context issues"
This reverts commit 3c15d578d4.

See https://github.com/NixOS/nixpkgs/issues/83863
2020-04-02 00:53:54 +02:00
Martin Milata
d4cbe042ef test-driver.py: use temporary dir for vde1.ctl
Send SIGTERM instead of SIGKILL to vde_switch to give it chance to
delete the directories.
2020-03-29 00:03:59 +01:00
worldofpeace
f59aa66fc1 make-iso9660-image.sh: enable joliet extension
This is per the advice of the osinfo-db maintainers https://gitlab.com/libosinfo/osinfo-db/-/merge_requests/107#note_313094852
2020-03-27 19:48:54 -04:00
Rouven Czerwinski
1685cd7fa7 nixos/test: update test-driver.py for mypy 0.770 2020-03-27 01:57:48 -07:00
Eelco Dolstra
4052f9b849
Compress optionsJSON using brotli 2020-03-24 16:42:04 +01:00
Silvan Mosberger
d28f138160
nixos: Fix pkgs exporting
This needs adjustment after dcdd232939
2020-03-19 21:19:25 +01:00
florianjacob
8b07500163
nixos/lib: use removePrefix in escapeSystemdPath 2020-03-14 03:13:04 +00:00
Michele Guerini Rocco
7b15d6cee4
Merge pull request #81241 from thefloweringash/nesting-system
nixos/activation: propagate system to nested configurations
2020-03-13 09:58:10 +01:00
Maximilian Bosch
6d14bac048
nixos/python-test-driver: allow single char as hostName and fix misleading error-message
In case of invalid chars, the error-message references "perl variables"
which is not the case here as the python-based framework is used.
2020-03-13 01:06:12 +01:00
Andrew Childs
ce416779bb nixos/activation: use eval-config's system argument for nesting
This avoids a possible surprise if the user is using `nixpkgs.system`
and `nesting.children`. `nesting.children` is expected to ignore all
parent configuration so we shouldn't propagate the user-facing option
`nixpkgs.system`. To avoid doing so, we introduce a new internal
option for holding the value passed to eval-config.nix, and use that
when recursing for nesting.
2020-03-05 20:28:31 +09:00
Jacek Galowicz
d772206c49 nixosTests: Make VLAN argument optional in python test driver 2020-02-14 19:05:03 +01:00
worldofpeace
95e41ecd8f
Merge pull request #79659 from worldofpeace/cleanup-python-test-lib
testing: fix runInMachineWithX/runInMachine
2020-02-14 12:38:57 -05:00
worldofpeace
aa4ba502d6
Merge pull request #79335 from flokli/deprecate-perl-vm-tests
nixos/testing: add deprecation notice for Perl VM tests
2020-02-09 21:08:11 -05:00
Florian Klink
0945178b3c nixos/testing: add deprecation notice for Perl VM tests
Most VM tests have been migrated to use the python test driver
(introduced in #71684), the migration is tracked in #72828 (which also
thankfully uncovered and fixed many currently broken tests)

While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.

We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.

Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.
2020-02-09 23:53:59 +01:00
worldofpeace
5507e09618 testing: fix runInMachineWithX 2020-02-09 16:27:01 -05:00
worldofpeace
88f76812f2 testing-python: readd auto displayManager
we import it for the runInMachineWithX
2020-02-09 16:23:35 -05:00
worldofpeace
fa9af83e96 testing-python: fix runInMachine
The test script's were unported.
It's unclear whether the preBuild or
postBuild will work as expect, due to
the linting of the test scripts.
2020-02-09 16:21:52 -05:00
Samuel Dionne-Riel
cebb0b49d4 Merge pull request #79368 from sorki/ext4resize
nixos/lib/make-ext4-fs: fall back to resize2fs -M if exact resize fails
2020-02-09 14:35:08 -05:00
Richard Marko
db894cb8ce nixos/lib/make-ext4-fs: fall back to resize2fs -M if exact resize fails
See also
- https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1415077/comments/4
- https://github.com/NixOS/nixpkgs/pull/62262
2020-02-09 17:18:21 +01:00
Jacek Galowicz
798fcaafa4 nixos/test: Add copy_from_host and copy_from_host_via_shell to python
test driver
2020-02-07 13:24:16 +01:00
John Ericson
14fbd4146d
Merge pull request #59855 from volth/qemu-cpu-passthru
runInLinuxVM, test-driver: pass host's cpu type to guest vm
2020-02-01 22:59:37 -05:00
Silvan Mosberger
80a2740991
Merge pull request #78265 from Synthetica9/https-homepages
treewide: fix redirected urls
2020-01-27 15:00:53 +01:00
Jacek Galowicz
f63ef280df nixosTests: Drop invalid utf characters in command output 2020-01-26 17:11:59 +01:00
Patrick Hilhorst
593e11fd94
treewide: fix redirected urls
According to https://repology.org/repository/nix_unstable/problems, we have a
lot of packages that have http links that redirect to https as their homepage.
This commit updates all these packages to use the https links as their
homepage.

The following script was used to make these updates:

```

curl https://repology.org/api/v1/repository/nix_unstable/problems \
    | jq '.[] | .problem' -r \
    | rg 'Homepage link "(.+)" is a permanent redirect to "(.+)" and should be updated' --replace 's@$1@$2@' \
    | sort | uniq > script.sed

find -name '*.nix' | xargs -P4 -- sed -f script.sed -i
```
2020-01-22 11:26:22 +01:00
Florian Klink
764aba4c1b
Merge pull request #77587 from singron/fix-shared
test-driver.py: Share the shared dir between VMs
2020-01-19 22:07:31 +01:00
Eric Culp
6795a4002c test-driver.py: Use /tmp/shared for copy_from_vm
The docstring says it uses a directory shared among all vms, although
that doesn't seem necessary for the functionality. However, it does need
to be consistent between the guest and host.
2020-01-19 10:33:46 -08:00
Florian Klink
ed0b5b6133
Merge pull request #77662 from marijanp/verbose-python-test-driver
nixos/test: added verbose output for failed tests
2020-01-17 13:50:49 +01:00
volth
2bd296a7e9 runInLinuxVM, test-driver: pass host's cpu type to guest vm
'kvm64' is the most generic CPU, which does not support SSE4.2, AVX and other ISA extentions.
2020-01-15 23:35:00 +00:00
Erik Arvstedt
5bdb653baf test-driver.py: fix decoding of VM output
The codec format 'unicode_escape' was introduced in 52ee102 to handle
undecodable bytes in boot menus.

This made the problem worse as unicode chars outside of iso-8859-1
produce garbled output and valid utf-8 strings (such as "\x" ) trigger
decoding errors.

Fix this by using the default 'utf-8' codec and by explicitly ignoring
decoding errors.
2020-01-14 15:52:44 -08:00
Erik Arvstedt
caa435fd1d test-driver.py: specify coreutils dependency
Otherwise the driver script fails when coreutils are not in PATH.
2020-01-14 15:52:44 -08:00
Marijan Petričević
61c61f80e5 nixos/test: added verbose output for failed tests 2020-01-14 09:33:10 +01:00
Eric Culp
21f567bdd9 test-driver.py: Share the shared dir between VMs
This changes the python test driver to match the behavior of the perl
test driver.  I.e. the directory mounted into /tmp/shared should be the
same for all machines.

This probably fixes many tests, but I found this while investigating
failures in nixos/tests/ceph-multi-node.nix.
2020-01-12 15:49:17 -08:00
Maximilian Bosch
b72661749b
nixos/python-test-driver: add an option to disable python linter
While it's a good idea to automate the linting of the python code used
for our tests, I think that it can be quite distracting when hacking on
a NixOS test.

I figured that it might be more convenient to add an option as a
shortcut for this to avoid that everyone needs to dig into the test
driver again.
2019-12-22 19:27:43 +01:00
Jan Tojnar
8dc5ff7dcf
nixos/displayManager: deprecate separate options for default wm/dm
The upstream session files display managers use have no concept of sessions being composed from
desktop manager and window manager. To be able to set upstream session files as default
session, we need a single option. Having two different ways to set default session would be confusing,
though, so we decided to deprecate the old method.

We also created separate script for each session, just like we already had a separate desktop
file for each one, and started using displayManager.sessionPackages mechanism to make the
session handling more uniform.
2019-12-15 04:16:20 +01:00
Bernardo Meurer
70c5a78062
nixos: compress make-ext4-fs with zstd 2019-12-13 16:18:04 -08:00
Jacek Galowicz
e05ae69904 nixos/test: Use retry() in all looping functions that need timeouts 2019-12-03 16:59:43 +01:00
Florian Klink
da3a320d13
Merge pull request #74077 from 7c6f434c/add-test-driver-py-copy-from-vm
Add test driver py copy from vm
2019-12-01 13:56:23 +01:00
Domen Kožar
2c8c19972d
Merge pull request #73966 from d-goldin/nixosoptions_xml_strctx
make-options-doc: fix string context issues
2019-11-25 13:20:00 +01:00
Michael Raskin
ad38a08ecb test-driver.py: add Machine.copy_from_vm 2019-11-25 11:35:48 +01:00
Michael Raskin
ef5bc381f9 test-driver.py: Fix the shared directory reference 2019-11-25 11:33:47 +01:00
Robin Gloster
ba03a1b731
nixos/tests: fail on failing subtests
This was the behaviour in the perl test driver too and a regression.

Also cleaned up unused imports
2019-11-25 01:08:06 +01:00
Dima
3c15d578d4 make-options-doc: fix string context issues
When using `documentation.nixos.includeAllModules = true;` with external
modules, the string context might contain dependencies to derivations
and so `toFile` refuses to evaluate;

```
error: in 'toFile': the file 'options.xml' cannot refer to derivation outputs, at
[...]/nixpkgs/nixos/lib/make-options-doc/default.nix:89:16
```

This is not an issue when using `writeText` (instead of manually
stripping the context).
2019-11-23 15:40:45 +01:00
Florian Klink
758efb9348
Merge pull request #73190 from flokli/ceph-tests-python
WIP: nixos/ceph: port tests to python
2019-11-21 20:13:41 +01:00
Jacek Galowicz
4dba4db1db nixos/test: let python driver evaluate test code in global scope so the
user can define and use functions
2019-11-20 19:08:18 +01:00
Jacek Galowicz
72dd1c6a74 nixos/test: If qemu monitor returns 0, simply return (like perl script does) 2019-11-20 17:38:20 +01:00
Jacek Galowicz
67e2042861 nixos/test: Fix child environment of python driver 2019-11-20 17:37:12 +01:00
worldofpeace
4583e293fd
Merge pull request #73251 from worldofpeace/remove-slim
nixos/slim: remove
2019-11-15 15:56:48 +00:00
worldofpeace
ce26b3eaf0 nixos/slim: remove
The SLIM project is abandoned and their last release was in 2013.
Because of this it poses a security risk to systems, no one is working
on it or picked up maintenance. It also lacks compatibility with systemd
and logind sessions. For users, there liikely isn't anything like slim
that's as lightweight in terms of dependencies.
2019-11-11 17:10:41 -05:00
Jörg Thalheim
03e6ca15e2
test-driver: add mypy support
It's a good idea to expand this in future to test code as well,
so we get type checking there as well.
2019-11-11 13:49:48 +00:00
Florian Klink
a8e9a93493 nixos/tests: fix succeed() with multiple commands
we previously immediately returned the first commands output, and didn't
execute any of the other commands.

Now, return the last commands output.
This should be documented in the method docstring.
2019-11-11 13:45:11 +01:00
Leonhard Markert
c7becac738
nixos/tests: correct arguments to get_unit_info 2019-11-08 15:01:42 +01:00
Marijan Petricevic
5854aaf680 nixos/tests: improve get_unit_info expressiveness on failure 2019-11-08 11:20:54 +01:00
worldofpeace
1e7ddf233a
Merge pull request #72943 from marijanp/port-test-driver-python
Port remaining test-driver functions to python
2019-11-07 20:23:27 +00:00
worldofpeace
6ed6d1af6a
Merge pull request #72947 from tfc/boot-test-port-fix
Boot test port fix
2019-11-07 17:32:43 +00:00
Jacek Galowicz
52ee1026b0 nixos/test: Handle undecodable bytes
This threw exceptions in boot menus
2019-11-07 10:00:39 +01:00
worldofpeace
3780b9e69c
Merge pull request #72835 from tfc/nixos-integration-test-ports
Nixos integration test ports
2019-11-07 01:05:36 +00:00
Marijan
9915a8ca45 nixos/tests: implemented python test-driver version of forwardPort 2019-11-06 22:43:37 +01:00
Marijan
01f79dfc9e nixos/tests: implemented python test-driver version of dumpTTYContents 2019-11-06 22:43:37 +01:00
Jacek Galowicz
cf138b4e6b nixos/tests: Fix subprocess launch call for VM monitor
This crashed in the create-script case
2019-11-06 21:48:08 +01:00
Daniel Schaefer
afbfef93a7 nixos/tests: Ignore shutdown/crash if not booted
Condition seems to be inverted. Crash and shutdown only make sense, when
the machine is booted; i.e. we return immediately otherwise.
In the Perl test driver this is:

    return unless $self->{booted};
2019-11-06 16:06:43 +01:00
Jacek Galowicz
45b339b736 nixos/tests: Implement python test-driver version of getWindowNames and waitForWindow 2019-11-05 18:07:05 +01:00
Jacek Galowicz
9c54658226 nixos/tests: Implement python test-driver version of requireActiveUnit 2019-11-05 18:07:00 +01:00
Florian Klink
ac97edf013 nixos/test: use ptpython as repl 2019-11-04 23:50:27 +01:00
Jacek Galowicz
3a28fefe7d nixos/test: Port test driver to python
Thanks @blitz and @jtraue for help with implementing machine methods
2019-11-04 23:50:27 +01:00
Chuck
ac65940537 nixos/make-iso9660-image: Fix doc typo 2019-10-18 11:58:28 -07:00
Chuck
4623f0a2c6 nixos/make-iso9660-image: Fix storeContents documentation
`packages` was renamed to `storeContents` in
668c146e33, but this comment
describing its purpose was not updated to match.
2019-10-18 11:58:05 -07:00
Eelco Dolstra
39c0e489d3
Revert "make-disk-image: replace nixos-install by nix-env calls"
This reverts commit e9bf955fd6. We use
nixos-install to ensure that make-disk-image produces the same result
as a regular installation (9802da517f)
and to reduce code duplication. If there is something broken in
nixos-install, it should be fixed there.
2019-10-07 17:21:28 +02:00
Christian Kögler
e9bf955fd6 make-disk-image: replace nixos-install by nix-env calls
nixos-install can only be called from a machine
2019-09-29 19:57:19 +02:00
Sarah Brofeldt
7ca5b39125 nixos/lib/utils: Make the set recursive again, unbreak eval 2019-09-08 13:31:18 +02:00