Commit Graph

621 Commits

Author SHA1 Message Date
Graham Christensen
bc49a0815a
utillinux: rename to util-linux 2020-11-24 12:42:06 -05:00
Frederik Rietdijk
f36b838e2a nixos test-driver: fix single line docstrings, fixes #104467
Single line docstrings should have the """ on a single line according to PEP 8.
It seems support for this landed in the latest version of Black.
2020-11-21 09:51:31 +01:00
Emery Hemingway
7e25b71132 nixos: use nativeBuildInputs in make- iso9660-image and system-tarball
The tools used to create iso9660 images and tarballs are independent of
the platform of the closure contained within.
2020-11-18 14:05:30 +01:00
Graham Christensen
d78aa080f5
make-disk-image: support legacy+gpt 2020-10-30 15:50:24 -04:00
Andreas Rammhold
eba8f5425f
nixos/tests: fix wrong inherit that passes on the nodes attrs
The hydra tarball step would fail due to the nodes attribute not being
properly inherited. Since we can't execute all the tests and release
steps locally anymore (thanks to the JSONification and faster hydra
eval) these errors will probably keep in appearing.

This is hopefully the last of those introduced by me test runner
refactoring.

Error was seen on hydra (https://hydra.nixos.org/build/129282411):
> unpacking sources
> unpacking source archive /nix/store/bp95x52h6nv3j8apxrryyj2rviw682k1-source
> source root is source
> patching sources
> autoconfPhase
> No bootstrap, bootstrap.sh, configure.in or configure.ac. Assuming this is not an GNU Autotools package.
> configuring
> release name is nixpkgs-21.03pre249116.1088f059401
> git-revision is 1088f05940
> building
> no Makefile, doing nothing
> running tests
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> checking Nixpkgs on i686-linux
> checking Nixpkgs on x86_64-linux
> checking Nixpkgs on x86_64-darwin
> checking eval-release.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> checking find-tarballs.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> error: while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:6:1, called from undefined position:
> while evaluating 'operator' at /build/source/maintainers/scripts/find-tarballs.nix:27:16, called from undefined position:
> while evaluating 'immediateDependenciesOf' at /build/source/maintainers/scripts/find-tarballs.nix:39:29, called from /build/source/maintainers/scripts/find-tarballs.nix:27:44:
> while evaluating anonymous function at /build/source/lib/attrsets.nix:234:10, called from undefined position:
> while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:40:37, called from /build/source/lib/attrsets.nix:234:16:
> while evaluating 'derivationsIn' at /build/source/maintainers/scripts/find-tarballs.nix:42:19, called from /build/source/maintainers/scripts/find-tarballs.nix:40:40:
> while evaluating 'canEval' at /build/source/maintainers/scripts/find-tarballs.nix:48:13, called from /build/source/maintainers/scripts/find-tarballs.nix:43:9:
> while evaluating the attribute 'nodes' at /build/source/nixos/lib/testing-python.nix:195:23:
> attribute 'nodes' missing, at /build/source/nixos/lib/testing-python.nix:193:16
> build time elapsed:  0m0.122s 0m0.043s 17m51.526s 0m56.668s
> builder for '/nix/store/96rk3c74vrk6m3snm7n6jhis3j640pn4-nixpkgs-tarball-21.03pre249116.1088f059401.drv' failed with exit code 1
2020-10-27 00:10:31 +01:00
Andreas Rammhold
73635b859d
nixos/tests: fix runInMachine
In 5500dc8 we introduced the --keep-vm-state flag and defaulted to that
flag not being set. This lead to the `runInMachine` tests not longer
working and that going unnoticed for quite some time now.
2020-10-25 20:09:33 +01:00
Andreas Rammhold
fa25d84d13
nixos/tests: fix testDriver reference in runInMachine function
In a previous commit I broke this as there is no longer one testDriver
but only a function to generate one based on some QEMU inputs.
2020-10-25 20:09:33 +01:00
Andreas Rammhold
61b09f552c
nixos/tests: format the testing-python.nix file more consistenly 2020-10-25 20:09:33 +01:00
Andreas Rammhold
04100cd281
nixos/tests: restructure test driver so that QEMU is actually overriden
Previously you would be able to override only the QEMU package to be
used in the test runner. Frankly that doesn't help a lot if you are
trying to get a graphical session. The graphical session requires the
option in the NixOS module system to bet set to the correct QEMU
package.

In this commit I moved most of the test node configuration and
transformations into the `mkDriver` function (previously called
`driver`). The motivation was to be able to create a `driver` instance
with a given QEMU package that will be used consistently througout the
test expression.
2020-10-25 20:09:33 +01:00
Konrad Borowski
254d30d4c9 test-driver.py: remove bufsize=1 from Popen calls
According to Python documentation [0], `bufsize=1` is only meaningful in
text mode. As we don't pass in an argument called `universal_newlines`,
`encoding`, `errors` or `text` the file objects aren't opened in text
mode, which means the argument is ignored with a warning in Python 3.8.

    line buffering (buffering=1) isn't supported in binary mode,
    the default buffer size will be used

This commit removes this warning that appared when using
interactive test driver built with `-A driver`. This is done by
removing `bufsize=1` from Popen calls.

The default parameter when unspecified for `bufsize` is `-1` which
according to the documentation will be interpreted as
`io.DEFAULT_BUFFER_SIZE`. As mentioned by a warning, Python already
uses default buffer size when providing `buffering=1` parameter for
file objects not opened in text mode.

[0]: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
2020-10-25 16:22:07 +01:00
rnhmjoj
bc2188b083
nixos: fix qemu_test being used in normal VMs
This is an attempt to fixup PR #49403.
2020-10-21 16:38:04 +02:00
Andreas Rammhold
20893b3a70
nixos/tests: expose both the interactive and non-interactive driver
For a lot of the work the non-interactive drivers are enough and it is
probably a good idea to keep it accessible for debugging without
touching the Nix expression.
2020-10-19 17:39:48 +02:00
Andreas Rammhold
c096880d46
nixos/tests: make the driver attribute use a rich qemu
Since we previously stripped down the features of `qemu_test` some of
the features users are used to while running tests through the (impure)
driver didn't work anymore. Most notably we lost support for graphical
output and audio. With this change the `driver` attribute uses are more
feature complete version of QEmu compared to the one used in the pure
Nix builds.

This gives us the best of both worlds. Users are able to see the
graphical windows of VMs while CI and regular nix builds do not have to
download all the (unnecessary) dependencies.
2020-10-19 17:39:48 +02:00
zowoq
008de9ca3c nixos/{containers,cri-o,podman}: move copyFile to nixos/lib/utils 2020-09-24 10:01:47 +10:00
Janne Heß
9f33ab62d9
nixos/testing: Add support for specialArgs
Since using flakes disallows the usage of <unstable> (which I use in
some tests), this adds an alternative. By setting specialArgs, all VMs
can get the `unstable` flake input as an arg. This is not possible with
extraConfigurations, as that would lead to infinite recursions.
2020-09-14 00:10:21 +02:00
Félix Baylac-Jacqué
e571b315e2
test-driver.py: defaulting keepVmState in Machine init
ecb73fd555 introduced a new keepVmState
CLI flag for test-driver.py. This CLI flags gets forwarded to the
Machine class through create_machine.

It created a regression for the boot tests where __main__ end up not
being evaluated. See
https://github.com/NixOS/nixpkgs/pull/97346#issuecomment-690951837 for
bug report.

Defaulting keepVmState to false when __main__ ends up not being
evaluated.
2020-09-11 10:47:03 +02:00
Félix Baylac-Jacqué
ecb73fd555
test-driver.py: fix VM state directory deletion
The previous version of the code would only kick in if the state
directory path pointed at a *file*, which never occurs. Making that
codepath actually work reveals an ordering bug, which this patch fixes
as well.

It also replaces the confusing, imperative case log message "delete VM
state directory" with "deleting VM state directory".

Finally, we hint the user about how to prevent this deletion. IE. by
passing the --keep-vm-state flag.

Bug report:
https://github.com/NixOS/nixpkgs/pull/91046#issuecomment-685568750

Credit goes to Edef for the rebase on top of a recent nixpkgs commit
and for writing most of this commit message.

Co-authored-by: edef <edef@edef.eu>
2020-09-07 12:26:40 +02:00
Florian Klink
98d6b55fdc nixos/testing: remove remaining coverage-data logic
This isn't used anymore as per
https://github.com/NixOS/nixpkgs/pull/72354#discussion_r451031449.
2020-09-05 16:07:59 +02:00
WORLDofPEACE
18348c7829
Merge pull request #96042 from rnhmjoj/loaOf
treewide: completely remove types.loaOf
2020-09-02 08:45:37 -04:00
rnhmjoj
3f8a3246f4
nixos/lib/make-options-doc: remove loaOf subs
Remove the substitution for the <name?> placeholder used by loaOf,
now that the type has been deprecated.
2020-09-02 00:42:51 +02:00
Anders Kaseorg
59b6664f15 Revert "Merge pull request #96254 from Mic92/logging"
This reverts commit 4fc708567f, reversing
changes made to 0e54f3a6d8.

Fixes #96699.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30 15:46:39 -07:00
Anders Kaseorg
a0a421bf5e Revert "Merge pull request #96152 from JJJollyjim/colour-test-machines-staging"
This reverts commit 1bff6fe17c, reversing
changes made to 2995fa48cb.

There’s presumably nothing wrong with this PR, except that it
conflicts with reverting #96254 which broke several tests (#96699).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30 15:45:44 -07:00
aszlig
b0ac24ae41
nixos/test-driver: Use guest time when using sleep
With the Perl driver, machine.sleep(N) was doing a sleep on the guest
machine instead of the host machine. The new Python test driver however
uses time.sleep(), which instead sleeps on the host.

While this shouldn't make a difference most of the time, it *does*
however make a huge difference if the test machine is loaded and you're
sleeping for a minimum duration of eg. an animation.

I stumbled on this while porting most of all my tests to the new Python
test driver and particularily my video game tests failed on a fairly
loaded machine, whereas they don't with the Perl test driver.

Switching the sleep() method to sleep on the guest instead of the host
fixes this.

Signed-off-by: aszlig <aszlig@nix.build>
2020-08-29 00:55:01 +02:00
Florian Klink
0620184f3f nixos/lib/test*: remove perl test driver
This has been deprecated in 20.03, and all tests have been migrated to
the python framework, effectively making this dead code.
2020-08-27 19:45:38 +02:00
Jamie McClymont
d7875caf76 nixos/test: colour machine names 2020-08-27 23:29:55 +12:00
Jörg Thalheim
4fc708567f
Merge pull request #96254 from Mic92/logging 2020-08-26 19:45:24 +01:00
Matthew Bauer
ca3fa9c32a
Merge pull request #95956 from matthewbauer/qemu-cpu-max
runInLinuxVM, test-driver: use -cpu max instead of -cpu host
2020-08-26 12:59:57 -05:00
Jörg Thalheim
87214dbd10
nixos/test-driver: re-introduce log()
Appearantly this is used in tests
2020-08-25 14:50:47 +01:00
Jörg Thalheim
f3c0a09c76
nixos/testdriver: sort imports 2020-08-25 10:15:24 +01:00
Jörg Thalheim
392415c285
nixos/test-driver: switch to pythons' logging lib
- Less code
- more thread-safe according to @flokli
2020-08-25 10:13:27 +01:00
Jörg Thalheim
c1667f85bb
nixos/test-driver: introduce main method
This way we not accidentally use introduce/use global variables.
Also it explictly mark the code for the mypy type checker.
2020-08-25 09:36:37 +01:00
Silvan Mosberger
b77d8ead28
Merge pull request #51850 from roberth/nixos-pure
nixos/lib/eval-config.nix: Add extraModules parameter for opt-in purity
2020-08-24 04:29:37 +02:00
Matthew Bauer
47b56e7c19 runInLinuxVM, test-driver: use -cpu max instead of -cpu host
This appears to avoid requiring KVM when it’s not available. This is
what I originally though -cpu host did. Unfortunately not much
documentation available from the QEMU side on this, but this appears
to square with help:

$ qemu-system-x86 -cpu help
...
x86 host                  KVM processor with all supported host features
x86 max                   Enables all features supported by the accelerator in the current host
...

Whether we actually want to support this not clear, since this only
happens when your CPU doesn’t have full KVM support. Some Nix builders
are lying about kvm support though. Things aren’t too slow without it
though.

Fixes https://github.com/NixOS/nixpkgs/issues/85394

Alternative to https://github.com/NixOS/nixpkgs/pull/83920
2020-08-21 23:42:07 -05:00
Janne Heß
ff03800d3b
nixos/testing: Fix fail() function
The docs say this behaves as succeed(), but it does not return stdout as
succeed() does. This fixes that behaviour
2020-08-21 21:28:24 +02:00
Jacek Galowicz
2a288cb1da
Merge pull request #93824 from blitz/fix-rpi4-installer
Fix Raspberry Pi 4B SD-Card Install Image
2020-08-08 13:45:02 +02:00
Maximilian Bosch
2ae62ef72f
Merge pull request #94611 from Ma27/rel-pkgs-error-msg
nixos/manual: improve error message for invalid values in `relatedPackages'
2020-08-03 20:17:46 +02:00
Maximilian Bosch
6d19c04416
nixos/manual: improve error message for invalid values in `relatedPackages'
As reported in NixOS discourse[1], tracking down invalid values in
`relatedPackages'[2] (i.e. list-items that don't exist in `pkgs`) is
fairly hard as the message "Invalid package attribute path `foobar'"
is quite unhelpful and the trace doesn't point to the source of the
problem either.

This patch improves the error message by mentioning that the issue is an
invalid `relatedPackages`-declaration in $optionName.

[1] https://discourse.nixos.org/t/invalid-package-attribute-path-nextcloud19/8403/9
[2] https://nixos.org/nixpkgs/manual/#sec-functions-library-options
2020-08-03 16:37:59 +02:00
zowoq
25d7880f17 nixos/lib/*: editorconfig fixes 2020-08-04 00:23:54 +10:00
Emery Hemingway
3f922834b8 nixos/lib/make-disk-image.nix: abritary format input
Pass unrecognized format types as the output file extension to
qemu-img. The motivation is support for "vdi" output.
2020-07-31 15:33:08 +02:00
Keshav Kini
5e86bba082 nixos/boot: some documentation improvements
- Give a more accurate description of how fileSystems.<name/>.neededForBoot
  works

- Give a more detailed description of how fileSystems.<name/>.encrypted.keyFile
  works
2020-07-29 14:39:21 -07:00
Lassulus
77cf1a6581
Merge pull request #89331 from Lassulus/make-disk-image
make-disk-image: add hybrid and dynamic sized images
2020-07-28 20:34:20 +02:00
lassulus
883a6079fd make-disk-image: add hybrid and dynamic sized images 2020-07-27 19:45:55 +02:00
Julian Stecklina
fd2047c642 nixos/lib/make-ext4-fs: fix after mkfs.ext4 refactoring
In 9ac1ab10c9 this library function was
refactored to use mkfs.ext4 instead of cptofs. There are two problems:

If populateImageCommands would create no files (which is fine), a cp
invocation would fail due to missing source arguments.

Another problem is that mkfs.ext4 relies on fakeroot to have sane
uid/gids in the generated filesystem image. This currently doesn't
work for cross compiling.
2020-07-26 22:49:55 +02:00
Timo Kaufmann
830a8d6ee1
Merge pull request #74174 from raboof/fix-74047-stable-gpt-disk-guid
make-iso9660-image: produce stable GPT disk GUID
2020-07-21 00:30:57 +02:00
Bas van Dijk
6e7822b8f3 lib: toHex -> toHexString & toBase -> toBaseDigits
This makes the type of these functions more apparent from the name.
2020-07-20 13:14:19 +02:00
Bas van Dijk
e15815e885 nixos/tests/networking.nix: test the services.dhcpd4.machines option
This modifies the `router` to not give out a range of IP addresses but
only give out a fixed address based on the MAC address using the
`services.dhcpd4.machines` option.

To get access to the MAC address the `qemuNicMac` function is defined
and exported from `qemu-flags.nix`.
2020-07-20 13:09:27 +02:00
Bas van Dijk
0410f5dff9 nixos/tests: support up to 255 nodes in NixOS tests 2020-07-20 13:09:27 +02:00
Arnout Engelen
be006eab1f
make-iso9660-image: produce stable GPT disk GUID
By generating a version-5 GUID based on $out (which contains
the derivation hash) and preventing isohybrid from overwriting
the GPT table (which already is populated correctly by xorriso).

Tested by:
* booting from USB disk on a UEFI system
* booting from USB disk on a non-UEFI system
* booting from CD on a UEFI system
* booting from CD on a non-UEFI system
* booting from CD on an OSX system

Also tested that "nix-build ./nixos/release-combined.nix -A
nixos.iso_minimal.x86_64-linux -I nixpkgs=~/nixpkgs-r13y --check"
now succeeds.

Fixes #74047
2020-07-20 11:16:59 +02:00
lassulus
8d05772134
nixos make-disk-image: fix permissions of /build
This was broken in 460c0d6 (PR #90431); now the nixos-unstable channel
should get unblocked.
vcunat modified this commit to use env-var instead of hardcoding /build
2020-07-10 10:31:26 +02:00
Justinas Stankevičius
f815cb6adf
nixos/test-driver: print a traceback when testScript fails (#92369)
* print a traceback: assertion message can be empty

* change print back to eprint
2020-07-06 23:29:40 +02:00
Samuel Dionne-Riel
736c7ca712
Merge pull request #82718 from misuzu/armv7l-ext4-fs-fix
nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs
2020-07-01 21:38:07 -04:00
misuzu
9ac1ab10c9 nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs
This fixes image creation on armv7l when image is bigger than 2G.
Also fix some reproducibility issues and other cptofs issues.
2020-07-01 11:32:28 +03:00
Michele Guerini Rocco
dab676b2d7
Merge pull request #65231 from buckley310/grub-password
grub: add support for passwords
2020-07-01 09:04:30 +02:00
Alexandre Esteves
e10e7d6a8b
testing-python: fix typo 2020-06-30 22:31:32 -05:00
Florian Klink
9e248c9ec9
Merge pull request #91046 from NinjaTrappeur/nin-delete-vm-state
test-driver.py: delete VM state directory after test run
2020-06-28 18:41:38 +02:00
rnhmjoj
b520055df6
nixos/lib/test-driver: add wait_for_console_text
This method is similar to wait_for_text but is based on matching
serial console lines instead of the VGA output.
2020-06-24 10:22:53 +02:00
Félix Baylac-Jacqué
7e7aa529d9
test-driver.py: delete VM state directory after test run
Keeping the VM state test across several run sometimes lead to subtle
and hard to spot errors in practice. We delete the VM state which
contains (among other things) the qcow volume.

We also introduce a -K (--keep-vm-state) flag making VM state to
persist after the test run. This flag makes test-driver.py to match
its previous behaviour.
2020-06-21 10:27:51 +02:00
Florian Klink
079de14fd3 nixos/make-ext4-fs: increase fudge factor from 1.03 to 1.10
Turns out, on smaller images (~800MiB uncompressed sdcard image size),
the current fudge factor is way too small to even get the system to the
phase where it can resize itself.

I first tried with 1.05, but it wasn't enough.
2020-06-21 09:47:02 +02:00
Peter Hoeg
eb62c7edc8
Merge pull request #90261 from prusnak/installer-zstd-rm
images: remove original files when using zstd for compression
2020-06-14 10:39:10 +08:00
Pavol Rusnak
490fc040be
images: remove original files when using zstd for compression 2020-06-14 00:19:11 +02:00
Emery Hemingway
b8a9e3eea1 nixos/lib/make-disk-image.nix: VDI support 2020-06-13 11:08:48 +05:30
Finn Behrens
f73436b374
nixos/make-iso9660: fix zstd input 2020-06-09 11:39:26 +02:00
Marek Mahut
7b9d7cc05d
Merge pull request #85947 from prusnak/images-zstd
Use zstd for ISO and SD images
2020-06-07 19:09:43 +02:00
Florian Klink
8a388c8296
Merge pull request #82258 from erikarvstedt/fix-xchg-caching
fix inconsistent caching of VM xchg dirs
2020-06-01 22:34:48 +02:00
Erik Arvstedt
d85dc4f690
qemu-vm: fix inconsistent caching of xchg dirs
xchg is advertised as a bidirectional exchange dir, but file content
transfer from host to VM fails due to caching:
If a file is read in the VM and then modified on the host, subsequent
re-reads in the VM can yield old, cached data.
This is caused by the use of 9p's cache=loose mode that is explicitly
meant for read-only mounts.

9p doesn't provide any suitable cache modes, so fix this by disabling
caching.

Also, remove a now unnecessary sync in the test driver.
2020-06-01 21:55:33 +02:00
Erik Arvstedt
dfa743e49e
test-driver: remove useless syncs
These syncs have the goal to transfer host filesystem changes to the VM,
but they have no effect because 1) syncing in the VM can't possibly pull
in host data and 2) 9p is accessing the host filesystem on the cached
layer anyways, so even syncing on the host would have no effect in the
VM.
2020-05-27 23:03:15 +02:00
Eelco Dolstra
0ffc85d64b
Remove unused files 2020-05-14 14:34:50 +02:00
Chuck
751a27020e nixos/test-driver: Specify /bin/sh shell when running a bourne shell script as the user
The test harness provides the commands it wishes to run in Bourne
syntax.  This fails if the user uses a different shell.  For example,
with fish:

  machine.wait_for_unit("graphical-session.target", "alice")

machine # fish: Unsupported use of '='. To run '-u`' with a modified environment, please use 'env XDG_RUNTIME_DIR=/run/user/`id -u`…'
machine # XDG_RUNTIME_DIR=/run/user/`id -u` systemctl --user --no-pager show "graphical-session.target"
machine # ^
machine # [   16.329957] su[1077]: pam_unix(su:session): session closed for user alice
error: retrieving systemctl info for unit "graphical-session.target" under user "alice" failed with exit code 127
2020-05-09 11:01:17 -07:00
Eelco Dolstra
3f80fadec4 testing{-python}.nix: Remove log pretty-printing cruft
This completes the removal of the nested log feature, which previously
got removed from Nix, Hydra, stdenv and GNU Make. In particular, this
means that the output of VM builds no longer contains a copy of
jQuery.
2020-05-07 15:56:30 +02:00
Eelco Dolstra
78f2a83029 test-driver.py: Fix deadlock when the log queue gets full
If a program (e.g. nixos-install) writes more than 1000 lines to
stderr during execute(), then process_serial_output() deadlocks
waiting for the queue to be processed. So use an unbounded queue
instead.

We should probably get rid of the structured log output (log.xml),
since then we don't need the log queue anymore.
2020-05-07 15:25:24 +02:00
Florian Klink
3cdd558a6f nixosTests: drop nr_tests and failed_tests variables
With the tests now bailing out early on a failing subtest, we don't need
to keep a list of failed tests, or the number of total tests
2020-05-01 01:22:42 +02:00
Jacek Galowicz
b139bc0d87 nixosTests: Reraise exception in subtests 2020-05-01 01:22:42 +02:00
Pavol Rusnak
f20fd89fc0
iso-image: use zstd for compression 2020-04-24 18:34:11 +02:00
Michael Reilly
84cf00f980
treewide: Per RFC45, remove all unquoted URLs 2020-04-10 17:54:53 +01:00
worldofpeace
a775961c8f Revert "Revert "iso-image: normalize volumeID""
This reverts commit 21b50e6f68.
2020-04-09 20:06:32 -04:00
worldofpeace
21b50e6f68
Revert "iso-image: normalize volumeID" 2020-04-09 19:43:59 -04:00
worldofpeace
8583d99e42
Merge pull request #83551 from worldofpeace/iso-id
iso-image: normalize volumeID
2020-04-09 19:31:52 -04:00
Silvan Mosberger
85e866db6f
nixos/lib/test-driver: Fix require_unit_state hardcoded formatting 2020-04-06 23:55:04 +02:00
Jan Tojnar
cab6b019b1
Revert "make-options-doc: fix string context issues"
This reverts commit 3c15d578d4.

See https://github.com/NixOS/nixpkgs/issues/83863
2020-04-02 00:53:54 +02:00
Martin Milata
d4cbe042ef test-driver.py: use temporary dir for vde1.ctl
Send SIGTERM instead of SIGKILL to vde_switch to give it chance to
delete the directories.
2020-03-29 00:03:59 +01:00
worldofpeace
f59aa66fc1 make-iso9660-image.sh: enable joliet extension
This is per the advice of the osinfo-db maintainers https://gitlab.com/libosinfo/osinfo-db/-/merge_requests/107#note_313094852
2020-03-27 19:48:54 -04:00
Rouven Czerwinski
1685cd7fa7 nixos/test: update test-driver.py for mypy 0.770 2020-03-27 01:57:48 -07:00
Eelco Dolstra
4052f9b849
Compress optionsJSON using brotli 2020-03-24 16:42:04 +01:00
Silvan Mosberger
d28f138160
nixos: Fix pkgs exporting
This needs adjustment after dcdd232939
2020-03-19 21:19:25 +01:00
florianjacob
8b07500163
nixos/lib: use removePrefix in escapeSystemdPath 2020-03-14 03:13:04 +00:00
Michele Guerini Rocco
7b15d6cee4
Merge pull request #81241 from thefloweringash/nesting-system
nixos/activation: propagate system to nested configurations
2020-03-13 09:58:10 +01:00
Maximilian Bosch
6d14bac048
nixos/python-test-driver: allow single char as hostName and fix misleading error-message
In case of invalid chars, the error-message references "perl variables"
which is not the case here as the python-based framework is used.
2020-03-13 01:06:12 +01:00
Andrew Childs
ce416779bb nixos/activation: use eval-config's system argument for nesting
This avoids a possible surprise if the user is using `nixpkgs.system`
and `nesting.children`. `nesting.children` is expected to ignore all
parent configuration so we shouldn't propagate the user-facing option
`nixpkgs.system`. To avoid doing so, we introduce a new internal
option for holding the value passed to eval-config.nix, and use that
when recursing for nesting.
2020-03-05 20:28:31 +09:00
Jacek Galowicz
d772206c49 nixosTests: Make VLAN argument optional in python test driver 2020-02-14 19:05:03 +01:00
worldofpeace
95e41ecd8f
Merge pull request #79659 from worldofpeace/cleanup-python-test-lib
testing: fix runInMachineWithX/runInMachine
2020-02-14 12:38:57 -05:00
worldofpeace
aa4ba502d6
Merge pull request #79335 from flokli/deprecate-perl-vm-tests
nixos/testing: add deprecation notice for Perl VM tests
2020-02-09 21:08:11 -05:00
Florian Klink
0945178b3c nixos/testing: add deprecation notice for Perl VM tests
Most VM tests have been migrated to use the python test driver
(introduced in #71684), the migration is tracked in #72828 (which also
thankfully uncovered and fixed many currently broken tests)

While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.

We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.

Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.
2020-02-09 23:53:59 +01:00
worldofpeace
5507e09618 testing: fix runInMachineWithX 2020-02-09 16:27:01 -05:00
worldofpeace
88f76812f2 testing-python: readd auto displayManager
we import it for the runInMachineWithX
2020-02-09 16:23:35 -05:00
worldofpeace
fa9af83e96 testing-python: fix runInMachine
The test script's were unported.
It's unclear whether the preBuild or
postBuild will work as expect, due to
the linting of the test scripts.
2020-02-09 16:21:52 -05:00
Samuel Dionne-Riel
cebb0b49d4 Merge pull request #79368 from sorki/ext4resize
nixos/lib/make-ext4-fs: fall back to resize2fs -M if exact resize fails
2020-02-09 14:35:08 -05:00
Richard Marko
db894cb8ce nixos/lib/make-ext4-fs: fall back to resize2fs -M if exact resize fails
See also
- https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1415077/comments/4
- https://github.com/NixOS/nixpkgs/pull/62262
2020-02-09 17:18:21 +01:00
Jacek Galowicz
798fcaafa4 nixos/test: Add copy_from_host and copy_from_host_via_shell to python
test driver
2020-02-07 13:24:16 +01:00
John Ericson
14fbd4146d
Merge pull request #59855 from volth/qemu-cpu-passthru
runInLinuxVM, test-driver: pass host's cpu type to guest vm
2020-02-01 22:59:37 -05:00
Silvan Mosberger
80a2740991
Merge pull request #78265 from Synthetica9/https-homepages
treewide: fix redirected urls
2020-01-27 15:00:53 +01:00
Jacek Galowicz
f63ef280df nixosTests: Drop invalid utf characters in command output 2020-01-26 17:11:59 +01:00
Patrick Hilhorst
593e11fd94
treewide: fix redirected urls
According to https://repology.org/repository/nix_unstable/problems, we have a
lot of packages that have http links that redirect to https as their homepage.
This commit updates all these packages to use the https links as their
homepage.

The following script was used to make these updates:

```

curl https://repology.org/api/v1/repository/nix_unstable/problems \
    | jq '.[] | .problem' -r \
    | rg 'Homepage link "(.+)" is a permanent redirect to "(.+)" and should be updated' --replace 's@$1@$2@' \
    | sort | uniq > script.sed

find -name '*.nix' | xargs -P4 -- sed -f script.sed -i
```
2020-01-22 11:26:22 +01:00
Florian Klink
764aba4c1b
Merge pull request #77587 from singron/fix-shared
test-driver.py: Share the shared dir between VMs
2020-01-19 22:07:31 +01:00
Eric Culp
6795a4002c test-driver.py: Use /tmp/shared for copy_from_vm
The docstring says it uses a directory shared among all vms, although
that doesn't seem necessary for the functionality. However, it does need
to be consistent between the guest and host.
2020-01-19 10:33:46 -08:00
Florian Klink
ed0b5b6133
Merge pull request #77662 from marijanp/verbose-python-test-driver
nixos/test: added verbose output for failed tests
2020-01-17 13:50:49 +01:00
volth
2bd296a7e9 runInLinuxVM, test-driver: pass host's cpu type to guest vm
'kvm64' is the most generic CPU, which does not support SSE4.2, AVX and other ISA extentions.
2020-01-15 23:35:00 +00:00
Erik Arvstedt
5bdb653baf test-driver.py: fix decoding of VM output
The codec format 'unicode_escape' was introduced in 52ee102 to handle
undecodable bytes in boot menus.

This made the problem worse as unicode chars outside of iso-8859-1
produce garbled output and valid utf-8 strings (such as "\x" ) trigger
decoding errors.

Fix this by using the default 'utf-8' codec and by explicitly ignoring
decoding errors.
2020-01-14 15:52:44 -08:00
Erik Arvstedt
caa435fd1d test-driver.py: specify coreutils dependency
Otherwise the driver script fails when coreutils are not in PATH.
2020-01-14 15:52:44 -08:00
Marijan Petričević
61c61f80e5 nixos/test: added verbose output for failed tests 2020-01-14 09:33:10 +01:00
Eric Culp
21f567bdd9 test-driver.py: Share the shared dir between VMs
This changes the python test driver to match the behavior of the perl
test driver.  I.e. the directory mounted into /tmp/shared should be the
same for all machines.

This probably fixes many tests, but I found this while investigating
failures in nixos/tests/ceph-multi-node.nix.
2020-01-12 15:49:17 -08:00
Maximilian Bosch
b72661749b
nixos/python-test-driver: add an option to disable python linter
While it's a good idea to automate the linting of the python code used
for our tests, I think that it can be quite distracting when hacking on
a NixOS test.

I figured that it might be more convenient to add an option as a
shortcut for this to avoid that everyone needs to dig into the test
driver again.
2019-12-22 19:27:43 +01:00
Jan Tojnar
8dc5ff7dcf
nixos/displayManager: deprecate separate options for default wm/dm
The upstream session files display managers use have no concept of sessions being composed from
desktop manager and window manager. To be able to set upstream session files as default
session, we need a single option. Having two different ways to set default session would be confusing,
though, so we decided to deprecate the old method.

We also created separate script for each session, just like we already had a separate desktop
file for each one, and started using displayManager.sessionPackages mechanism to make the
session handling more uniform.
2019-12-15 04:16:20 +01:00
Bernardo Meurer
70c5a78062
nixos: compress make-ext4-fs with zstd 2019-12-13 16:18:04 -08:00
Jacek Galowicz
e05ae69904 nixos/test: Use retry() in all looping functions that need timeouts 2019-12-03 16:59:43 +01:00
Florian Klink
da3a320d13
Merge pull request #74077 from 7c6f434c/add-test-driver-py-copy-from-vm
Add test driver py copy from vm
2019-12-01 13:56:23 +01:00
Domen Kožar
2c8c19972d
Merge pull request #73966 from d-goldin/nixosoptions_xml_strctx
make-options-doc: fix string context issues
2019-11-25 13:20:00 +01:00
Michael Raskin
ad38a08ecb test-driver.py: add Machine.copy_from_vm 2019-11-25 11:35:48 +01:00
Michael Raskin
ef5bc381f9 test-driver.py: Fix the shared directory reference 2019-11-25 11:33:47 +01:00
Robin Gloster
ba03a1b731
nixos/tests: fail on failing subtests
This was the behaviour in the perl test driver too and a regression.

Also cleaned up unused imports
2019-11-25 01:08:06 +01:00
Dima
3c15d578d4 make-options-doc: fix string context issues
When using `documentation.nixos.includeAllModules = true;` with external
modules, the string context might contain dependencies to derivations
and so `toFile` refuses to evaluate;

```
error: in 'toFile': the file 'options.xml' cannot refer to derivation outputs, at
[...]/nixpkgs/nixos/lib/make-options-doc/default.nix:89:16
```

This is not an issue when using `writeText` (instead of manually
stripping the context).
2019-11-23 15:40:45 +01:00
Florian Klink
758efb9348
Merge pull request #73190 from flokli/ceph-tests-python
WIP: nixos/ceph: port tests to python
2019-11-21 20:13:41 +01:00
Jacek Galowicz
4dba4db1db nixos/test: let python driver evaluate test code in global scope so the
user can define and use functions
2019-11-20 19:08:18 +01:00
Jacek Galowicz
72dd1c6a74 nixos/test: If qemu monitor returns 0, simply return (like perl script does) 2019-11-20 17:38:20 +01:00
Jacek Galowicz
67e2042861 nixos/test: Fix child environment of python driver 2019-11-20 17:37:12 +01:00
worldofpeace
4583e293fd
Merge pull request #73251 from worldofpeace/remove-slim
nixos/slim: remove
2019-11-15 15:56:48 +00:00
worldofpeace
ce26b3eaf0 nixos/slim: remove
The SLIM project is abandoned and their last release was in 2013.
Because of this it poses a security risk to systems, no one is working
on it or picked up maintenance. It also lacks compatibility with systemd
and logind sessions. For users, there liikely isn't anything like slim
that's as lightweight in terms of dependencies.
2019-11-11 17:10:41 -05:00
Jörg Thalheim
03e6ca15e2
test-driver: add mypy support
It's a good idea to expand this in future to test code as well,
so we get type checking there as well.
2019-11-11 13:49:48 +00:00
Florian Klink
a8e9a93493 nixos/tests: fix succeed() with multiple commands
we previously immediately returned the first commands output, and didn't
execute any of the other commands.

Now, return the last commands output.
This should be documented in the method docstring.
2019-11-11 13:45:11 +01:00
Leonhard Markert
c7becac738
nixos/tests: correct arguments to get_unit_info 2019-11-08 15:01:42 +01:00
Marijan Petricevic
5854aaf680 nixos/tests: improve get_unit_info expressiveness on failure 2019-11-08 11:20:54 +01:00
worldofpeace
1e7ddf233a
Merge pull request #72943 from marijanp/port-test-driver-python
Port remaining test-driver functions to python
2019-11-07 20:23:27 +00:00
worldofpeace
6ed6d1af6a
Merge pull request #72947 from tfc/boot-test-port-fix
Boot test port fix
2019-11-07 17:32:43 +00:00
Jacek Galowicz
52ee1026b0 nixos/test: Handle undecodable bytes
This threw exceptions in boot menus
2019-11-07 10:00:39 +01:00
worldofpeace
3780b9e69c
Merge pull request #72835 from tfc/nixos-integration-test-ports
Nixos integration test ports
2019-11-07 01:05:36 +00:00
Marijan
9915a8ca45 nixos/tests: implemented python test-driver version of forwardPort 2019-11-06 22:43:37 +01:00
Marijan
01f79dfc9e nixos/tests: implemented python test-driver version of dumpTTYContents 2019-11-06 22:43:37 +01:00
Jacek Galowicz
cf138b4e6b nixos/tests: Fix subprocess launch call for VM monitor
This crashed in the create-script case
2019-11-06 21:48:08 +01:00
Daniel Schaefer
afbfef93a7 nixos/tests: Ignore shutdown/crash if not booted
Condition seems to be inverted. Crash and shutdown only make sense, when
the machine is booted; i.e. we return immediately otherwise.
In the Perl test driver this is:

    return unless $self->{booted};
2019-11-06 16:06:43 +01:00
Jacek Galowicz
45b339b736 nixos/tests: Implement python test-driver version of getWindowNames and waitForWindow 2019-11-05 18:07:05 +01:00
Jacek Galowicz
9c54658226 nixos/tests: Implement python test-driver version of requireActiveUnit 2019-11-05 18:07:00 +01:00
Florian Klink
ac97edf013 nixos/test: use ptpython as repl 2019-11-04 23:50:27 +01:00
Jacek Galowicz
3a28fefe7d nixos/test: Port test driver to python
Thanks @blitz and @jtraue for help with implementing machine methods
2019-11-04 23:50:27 +01:00
Chuck
ac65940537 nixos/make-iso9660-image: Fix doc typo 2019-10-18 11:58:28 -07:00
Chuck
4623f0a2c6 nixos/make-iso9660-image: Fix storeContents documentation
`packages` was renamed to `storeContents` in
668c146e33, but this comment
describing its purpose was not updated to match.
2019-10-18 11:58:05 -07:00
Eelco Dolstra
39c0e489d3
Revert "make-disk-image: replace nixos-install by nix-env calls"
This reverts commit e9bf955fd6. We use
nixos-install to ensure that make-disk-image produces the same result
as a regular installation (9802da517f)
and to reduce code duplication. If there is something broken in
nixos-install, it should be fixed there.
2019-10-07 17:21:28 +02:00
Christian Kögler
e9bf955fd6 make-disk-image: replace nixos-install by nix-env calls
nixos-install can only be called from a machine
2019-09-29 19:57:19 +02:00
Sarah Brofeldt
7ca5b39125 nixos/lib/utils: Make the set recursive again, unbreak eval 2019-09-08 13:31:18 +02:00
Florian Klink
2f3b9cd52c
Merge pull request #66274 from talyz/gitlab
nixos/gitlab: Add support for secure secrets and more
2019-09-07 12:52:44 -07:00
talyz
64358cb0e9 nixos/utils: Handle arbitrary secrets in JSON output files
Introduce new functions which allows modules to define options where,
if the input is an attrset and the output is JSON, the user can define
arbitrary secrets.
2019-09-06 16:56:25 +02:00
volth
08f68313a4 treewide: remove redundant rec 2019-08-28 11:07:32 +00:00
Frederik Rietdijk
5061fe0c2c Merge staging-next into staging 2019-08-28 08:26:42 +02:00
volth
35d68ef143 treewide: remove redundant quotes 2019-08-26 21:40:19 +00:00
Clemens Fruhwirth
f664823a1a make-disk-image.nix: Handle cptofs error explicitly
Because the copy process inside the VM does not reliably
give "No space" error message leaving the user wondering what
went wrong:

unable to create directory /mnt/0000fe01///nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale/ro: Success
[    5.462365] reboot: Restarting system
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale/ro, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7, aborting
error processing entry /build/root/nix/store, aborting
error processing entry /build/root/nix, aborting
builder for '/nix/store/fsdvqxq92iai7f3w8wcsncgfwag7cj2l-libvirtd-ssh-image.drv' failed with exit code 228
2019-08-26 13:09:53 +02:00
danbst
d80cd26ff9 Merge branch 'master' into flip-map-foreach 2019-08-18 18:00:25 +03:00
Domen Kožar
dcd50c0ea0
pkgs.lib -> lib 2019-08-12 11:46:53 +02:00
Domen Kožar
6cf861c617
make-options-doc: add asciidoc variant 2019-08-10 20:11:04 +02:00
Domen Kožar
3a93fcfd1e
make-options-doc: expose Nix set 2019-08-10 14:24:11 +02:00
Domen Kožar
5cfd034af0
Extract NixOS options documentation generation to a function
Motivation is to support other repositories containing nixos
modules that would like to generate options documentation:

- nix-darwin
- private repos
- arion
- ??
2019-08-08 16:18:09 +02:00
Nikola Knezevic
d0ef94258d Make hostname in tests overridable
The original form effectively forbade any NixOS configuration that is under
test to explicitly set the hostname.
2019-08-07 10:31:13 +02:00
Danylo Hlynskyi
7585496eff
Merge branch 'master' into flip-map-foreach 2019-08-05 14:09:28 +03:00
danbst
0f8596ab3f mass replace "flip map -> forEach"
See `forEach`-introduction commit.
```
rg 'flip map ' --files-with-matches | xargs sed -i 's/flip map /forEach /g'
```
2019-08-05 14:03:38 +03:00
danbst
91bb646e98 Revert "mass replace "flip map -> foreach""
This reverts commit 3b0534310c.
2019-08-05 14:01:45 +03:00
Andrew Childs
d2144755a4 nixos-test-driver: allow configuration of net frontend and backend
When IPXE tests were added, an option was added for configuring only
the frontend, and the backend configuration was dropped entirely. This
caused most installer tests to fail.
2019-07-22 13:44:27 +03:00
Nikolay Amiantov
81d35a9d7e nixos-test-driver: support netRomFile
Needed for UEFI PXE netboot testing.
2019-07-15 19:33:26 +03:00
danbst
3b0534310c mass replace "flip map -> foreach"
See `foreach`-introduction commit.
```
rg 'flip map ' --files-with-matches | xargs sed -i 's/flip map /foreach /g'
```
2019-07-14 13:46:10 +03:00
worldofpeace
3f4a353737 treewide: use dontUnpack 2019-07-01 04:23:51 -04:00
Samuel Dionne-Riel
a0aff40f65 make-ext4-fs: Allows populating with custom files
This will allow adding /boot files to the rootfs.
2019-06-16 17:47:30 -04:00
Arian van Putten
cbc45b5981 nixos: Make nesting.children work in NixOS tests
We differentiate between modules and baseModules in  the
VM builder for NixOS tests. This way, nesting.children, eventhough
it doesn't inherit from parent, still has enough config to
actually complete the test. Otherwise, the qemu modules
would not be loaded, for example, and a nesting.children
statement would not evaluate.
2019-05-29 12:50:49 +02:00
Arian van Putten
d50b434234 nixos: Make 'nesting.clone' work in NixOS tests
Because nesting.clone calls 'eval-config.nix' manually,
without the 'extraArgs' argument that provides the 'nodes'
argument to nixos modules in nixos tests, evaluating
of 'nesting.clone' definitions would fail with the following error

while evaluating the module argument `nodes' in "<redacted>"
while evaluating the attribute '_module.args.nodes' at undefined position:
attribute 'nodes' missing, at <redacted./nixpkgs/lib/modules.nix:163:28

by not using 'extraArgs' but a nixos module instead, the nodes parameter
gets propagated to the 'eval-config.nix' call that  'nesting.clone'
makes too -  getting rid of the error.

See  https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/activation/top-level.nix#L13-L23
See  https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/build-vms.nix#L27
See  https://github.com/NixOS/nixpkgs/issues/20886#issuecomment-495952149
2019-05-26 00:20:21 +02:00
Arian van Putten
2e75a7b516 nixos: doc: optionally include all modules in manual generation
Before this change `man 5 configuration.nix` would only show options of modules in
the `baseModules` set, which consists only of the list of modules in
`nixos/modules/module-list.nix`

With this change applied and `documentation.nixos.includeAllModules` option enabled
all modules included in `configuration.nix` file will be used instead.

This makes configurations with custom modules self-documenting. It also means
that importing non-`baseModules` modules like `gce.nix` or `azure.nix`
will make their documentation available in `man 5 configuration.nix`.

`documentation.nixos.includeAllModules` is currently set to `false` by
default as enabling it usually uncovers bugs and prevents evaluation.
It should be set to `true` in a release or two.

This was originally implemented in #47177, edited for more configurability,
documented and rebased onto master by @oxij.
2019-03-05 09:41:40 +00:00
Matthew Bauer
878965731f
nixos/lib/testing.nix: config defaults to {}
Fixes #51858

56e12aae54 ends up passing config to pkgs. Unfortunately this might be null and pkgs/top-level/default.nix assumes it is an attrset. To fix this, we just make the default for config = {}. Thanks to @kristoff3r for tracking this down.

/cc @domenkozar
2019-02-06 20:04:24 -05:00
Travis Athougies
a66ef3aa3d Add options to build disk image function (#50239)
* add options to build disk image function

* Revert suffix changes
2019-01-24 01:47:55 +02:00
Samuel Dionne-Riel
50555a6d35
Merge pull request #54330 from samueldr/fix/disable-tests-shell-access
Revert "Add ssh backdoor to VM tests infrastructure."
2019-01-19 14:20:15 -05:00
Samuel Dionne-Riel
3aab228d09 Revert "Add ssh backdoor to VM tests infrastructure."
This reverts commit d6e3db44cf.

See #53935 for explanations. In short, it may be causing issues with
tests on the build infrastructure.
2019-01-19 13:24:39 -05:00
Danylo Hlynskyi
23a13b562c
kernel config: add support for CONFIG_SQUASHFS_ZSTD (#52967)
Also, allow override `make-squashfs.nix` compression parameters.
2019-01-17 15:24:44 +02:00
Silvan Mosberger
497e6f1705
Merge pull request #51661 from eonpatapon/testing-hosts
test: set machines fqdn in /etc/hosts
2019-01-12 20:41:13 +01:00
Samuel Dionne-Riel
2646a64fbc
Merge pull request #53827 from samueldr/feature/data-in-logs
tests: Logs timing in tests
2019-01-12 12:54:56 -05:00
Samuel Dionne-Riel
3b68ddb6fe
Merge pull request #53828 from samueldr/feature/double-alarm-time
tests: Wait for shell for twice as long (10m)
2019-01-12 12:08:14 -05:00
Samuel Dionne-Riel
b28b37eb00 tests: Wait for shell for twice as long (10m)
See #49441 for an earlier attempt, which was subsequently reverted. I am
assuming that doubling the time will be sufficient if the machine is
overloaded since so many of the tests already pass at 5 minutes, while
still not holding back failures for needlessly long.
2019-01-11 22:40:19 -05:00
Samuel Dionne-Riel
5d93e2c01c test-driver: Logs time taken for nests 2019-01-11 22:36:31 -05:00
Samuel Dionne-Riel
1fe0018df8 test-driver: Adds time it took to connect to guest in logs
This will make it possible to track whether the time is generous or not
when ran on hydra.
2019-01-11 22:36:31 -05:00
Matthew Bauer
04373fd3cc
Merge pull request #52594 from matthewbauer/fix-51025
make-disk-image: use filterSource instead of cleanSource
2019-01-07 16:29:58 -06:00
Matthew Bauer
f05d8f31ec make-disk-image: use filterSource instead of cleanSource
cleanSource does not appear to work correctly in this case. The path
does not get coerced to a string, resulting in a dangling symlink
produced in channel.nix.  Not sure why, but this
seems to fix it.

Fixes #51025.

/cc @elvishjericco
2019-01-07 16:28:50 -06:00
Frederik Rietdijk
9618abe87c Merge master into staging-next 2019-01-04 21:13:19 +01:00
Matthew Bauer
74312c7ef5
Merge pull request #52760 from akru/master
lib/make-ext4-fs: more efficient store maker
2019-01-03 15:07:27 -06:00
Alexander Krupenkin
2f0c495c31
lib/make-ext4-fs: more efficient store maker 2018-12-24 23:21:15 +03:00
Jan Tojnar
ef935fa101
Merge branch 'master' into staging 2018-12-24 15:02:29 +01:00
msteen
8d217ede58 fix infinite recursion caused by the unnecessary inspection of options + fix is parent of mount point check (#51541) 2018-12-24 14:05:55 +01:00
Michael Raskin
ede54f9144
Merge pull request #52379 from erikarvstedt/tesseract
Major tesseract improvements
2018-12-20 14:41:48 +00:00
Erik Arvstedt
8d1ba999cb
tesseract: rename to tesseract4, add alias
This is more consistent with the naming of the most popular versioned pkgs.
2018-12-19 18:09:56 +01:00
Frederik Rietdijk
9ab61ab8e2 Merge staging-next into staging 2018-12-19 09:00:36 +01:00
Maximilian Bosch
6c6341335b
nixos/test-driver: fix wording in error message about invalid node names
Since 113a6b9325 the test driver
explicitly ensures if the node names won't break the resulting Perl
script at runtime. This slightly improves the correctness of the error
message.
2018-12-18 23:46:54 +01:00
Franz Pletz
670c5ac8ef
Merge pull request #46806 from Ma27/disallow-dash-separators-in-machine-declarations
nixos/testing: disallow special chars in machine names in network expressions
2018-12-18 01:03:34 +00:00
Maximilian Bosch
113a6b9325
nixos/testing: disallow special chars in machine names in network expressions
These names are referenced by Perl variables inside the testing
frameworks which don't allow chars like `-` as character inside. An exemplary
expression may look like this:

```
{
  x11-vm = {
    services.xserver.enable = true;
  };
}
```

This expression evaluates, e.g. when running `nixos-build-vms`, but when
trying to run `./result/bin/nixos-run-vms`, an error like this occurs:

```
starting VDE switch for network 1
running the VM test script
error: Can't modify subtraction (-) in scalar assignment at (eval 17) line 1, at EOF
Bareword "test" not allowed while "strict subs" in use at (eval 17) line 1.
Can't modify subtraction (-) in scalar assignment at (eval 17) line 1, at EOF
Bareword "test" not allowed while "strict subs" in use at (eval 17) line 1.
vde_switch: EOF on stdin, cleaning up and exiting
cleaning up
```

This can be very confusing for beginners, this change breaks evaluation
if such names are used for machines.
2018-12-18 01:58:56 +01:00
volth
bb9557eb7c lib.makePerlPath -> perlPackages.makePerlPath 2018-12-15 03:50:31 +00:00
Samuel Dionne-Riel
ad27b068d7 installer: Adds AArch64 UEFI installer support. 2018-12-12 19:26:34 -05:00
Robert Hensing
4272188364 nixos/lib/eval-config.nix: Add extraModules parameter for opt-in purity 2018-12-10 22:02:57 +01:00
Jean-Philippe Braun
764f16461b test: set machines fqdn in /etc/hosts 2018-12-07 14:22:21 +01:00
Samuel Dionne-Riel
70488665fa
Merge pull request #51207 from samueldr/fix/sd-image-slimming
sd-image: Slims the ext4 filesystem even more.
2018-12-06 23:35:09 +00:00
Samuel Dionne-Riel
61bdaad9a9 sd-image: Slims the ext4 filesystem even more.
This is to try and squeeze more lost space from the image, so that hydra
starts building it again.

The fsck previous to the resize2fs is required so resize2fs works.

The one afterwards is a sanity check.

Using `-M` from resize2fs will not give much saved space due to a known
(in the manual) issue.

```
[samueldr@aarch64:~/nixpkgs]$ ls -lh result-*/*/*.img
-r--r--r-- 1 root root 2.2G Jan  1  1970 result-original/sd-image/nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img
-r--r--r-- 1 root root 2.1G Jan  1  1970 result-M/sd-image/nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img
-r--r--r-- 1 root root 1.9G Jan  1  1970 result-slimmed/sd-image/nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img
```

```
[samueldr@aarch64:~/nixpkgs]$ nix path-info -S ./result-original
/nix/store/c8k9n78gylx293rjh762fr05a069kxp2-nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img   3844125000

[samueldr@aarch64:~/nixpkgs]$ nix path-info -S ./result-slimmed
/nix/store/962238skj5mnzhrsmjy23dyzmxk77sp4-nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img   3447473208
```
2018-11-30 19:11:49 -05:00
Domen Kožar
56e12aae54
Fix nixops evaluation
See https://discourse.nixos.org/t/nixops-stopped-working/1527
2018-11-26 23:12:31 +00:00
Ding Xiang Fei
b011049cf6 Merge branch 'master' of https://github.com/nixos/nixpkgs into tarball-closureinfo 2018-11-26 12:04:07 +08:00
Ding Xiang Fei
ceececbd04 Merge branch 'master' of https://github.com/nixos/nixpkgs into tarball-closureinfo 2018-11-14 12:32:28 +08:00
Robert Hensing
cdca66d7e8 Add pkgs.nixosTest 2018-11-13 10:43:05 +01:00
Léo Gaspard
cae5598611
meta.tests: drop meta.needsVMSupport
Its job is already handled by `requiredSystemFeatures`
2018-11-11 23:11:46 +09:00
Léo Gaspard
6c68fbd4e1
tests: refactor to carry the package set as an argument
This way, the package set will be possible to pass without re-importing
all the time
2018-11-11 23:11:45 +09:00
Ding Xiang Fei
4259f7575e use closure-info for building system tarball 2018-11-07 12:52:53 +08:00
Timo Kaufmann
6141939d6e
Merge pull request #44439 from Ekleog/meta-tests
[RFC] Use `meta.tests` to link from packages to the tests that test them
2018-11-07 00:05:22 +01:00
Sarah Brofeldt
95486ca306 Revert "NixOS tests: Wait for shell for 10x longer (50m)"
This reverts commit 9bc10e1291.
2018-11-05 08:58:08 +00:00
Matthew Bauer
d3c2247b77
Merge pull request #47679 from NixOS/docs-make-channel
Document make-channel.nix
2018-11-04 16:51:28 -06:00
Robert Hensing
17e863d017 Remove incorrect sentence from make-channel comment
Besides being incorrect, claims like these tend to stick around
after they become incorrect.
2018-11-04 20:45:49 +01:00
Léo Gaspard
02e1f00ffd
dovecot, opensmtpd: add link to test in meta.tests
Rationale
---------

Currently, tests are hard to discover. For instance, someone updating
`dovecot` might not notice that the interaction of `dovecot` with
`opensmtpd` is handled in the `opensmtpd.nix` test.

And even for someone updating `opensmtpd`, it requires manual work to go
check in `nixos/tests` whether there is actually a test, especially
given not so many packages in `nixpkgs` have tests and this is thus most
of the time useless.

Finally, for the reviewer, it is much easier to check that the “Tested
via one or more NixOS test(s)” has been checked if the file modified
already includes the list of relevant tests.

Implementation
--------------

Currently, this commit only adds the metadata in the package. Each
element of the `meta.tests` attribute is a derivation that, when it
builds successfully, means the test has passed (ie. following the same
convention as NixOS tests).

Future Work
-----------

In the future, the tools could be made aware of this `meta.tests`
attribute, and for instance a `--with-tests` could be added to
`nix-build` so that it also builds all the tests. Or a `--without-tests`
to build without all the tests. @Profpatsch described in his NixCon talk
such systems.

Another thing that would help in the future would be the possibility to
reasonably easily have cross-derivation nix tests without the whole
NixOS VM stack. @7c6f434c already proposed such a system.

This RFC currently handles none of these concerns. Only the addition of
`meta.tests` as metadata to be used by maintainers to remember to run
relevant tests.
2018-10-30 21:31:39 +09:00
Sarah Brofeldt
9bc10e1291 NixOS tests: Wait for shell for 10x longer (50m) 2018-10-30 09:22:42 +01:00
Eelco Dolstra
0d15004cba
Merge pull request #49401 from aherrmann/stringify-modules-path
nixos/lib/eval-config.nix: toString modulesPath
2018-10-29 16:21:09 +01:00
Andreas Herrmann
044ceae280 nixos/lib/eval-config.nix: toString modulesPath
Referencing modulesPath in NixOS configurations can cause evaluation
errors in restricted mode.  If used as `${modulesPath}` (as in all
use-sites in nixpkgs) the modules subtree is copied into its own store
path. Access to this path will be forbidden in restricted mode.

Converting to a string solves this issue.
`${builtins.toString modulesPath}` will point to a subdirectory of the
nixpkgs tree out of which evalModules is called.

This change converts modulesPath to a string by default so that the
call-site doesn't have to anymore.
2018-10-29 15:46:20 +01:00
Linus Heckemann
bbffdfcf72 Document make-channel.nix 2018-10-02 21:52:43 +02:00
Tuomas Tynkkynen
d6e3db44cf Add ssh backdoor to VM tests infrastructure.
Thanks to @dezgeg for prototype implementation, I've
cleaned it up and added documentation.
2018-09-28 10:53:08 +01:00
Jan Malakhovski
0f3b89bbed nixos: doc: move non-service parts of service.nixosManual to documentation.nixos 2018-09-23 20:50:47 +00:00
Graham Christensen
ca7391daf2
Merge pull request #46341 from obsidiansystems/fix-46320
nixpkgs module: Fix defaulting of `localSystem` and `system`
2018-09-08 09:16:41 -04:00
John Ericson
9f9723b179 nixpkgs module: Fix defaulting of localSystem and system
Take two of #40708 (4fe2898608).

That PR attempted to bidirectionally default `config.nixpkgs.system` and
`config.nixpkgs.localSystem.system` to each be updated by the other. But
this is not possible with the way the module system works. Divergence in
certain cases in inevitable.

This PR is more conservative and just has `system` default `localSystem`
and `localSystem` make the final call as-is. This solves a number of
issues.

 - `localSystem` completely overrides `system`, just like with nixpkgs
 proper. There is no need to specify `localSystem.system` to clobber the
 old system.

 - `config.nixpkgs.localSystem` is exactly what is passed to nixpkgs. No
 spooky steps.

 - `config.nixpkgs.localSystem` is elaborated just as nixpkgs would so
 that all attributes are available, not just the ones the user
 specified.

The remaining issue is just that `config.nixpkgs.system` doesn't update
based on `config.nixpkgs.localSystem.system`. It should never be
referred to lest it is a bogus stale value because
`config.nixpkgs.localSystem` overwrites it.

Fixes #46320
2018-09-07 16:43:56 -04:00
Shea Levy
18337f3ece
Merge branch 'no-toPath' 2018-09-06 08:09:53 -04:00
John Ericson
2c2f1e37d4 reewide: Purge all uses stdenv.system and top-level system
It is deprecated and will be removed after 18.09.
2018-08-30 17:20:32 -04:00
xeji
b2dc75cd03
Merge pull request #43736 from volth/patch-208
qemu: 2.12.1 -> 3.0.0
2018-08-26 01:28:12 +02:00
Will Fancher
47d2f92a05 make-ext4-fs: Fix cross compiling 2018-08-21 14:05:23 +03:00
volth
341250fa10 qemu: 2.12.1 -> 3.0.0 2018-08-20 22:02:02 +00:00
volth
2e979e8ceb [bot] nixos/*: remove unused arguments in lambdas 2018-07-20 20:56:59 +00:00
Andrew Dunham
d7bfd04301 sdImage: make partition ID/UUID configurable 2018-07-12 17:02:03 +03:00
Matthew Bauer
753c63dbc1 qemu-flags: default to qemu-kvm
this is the "generic" version & should work everywhere.

also add darwin case
2018-05-31 12:34:15 -04:00
Shea Levy
6da6accd30
treewide: Remove uses of builtins.toPath.
toPath has confusing semantics and is never necessary; it can always
either just be omitted or replaced by pre-concatenating `/.`. It has
been marked as "!!! obsolete?" for more than 10 years in a C++
comment, hopefully removing it will let us properly deprecate and,
eventually, remove it.
2018-05-22 16:42:02 -04:00
Tuomas Tynkkynen
bc828721ed nixos/lib/make-ext4-fs: Add a sanity check
I ended up with a corrupted image with the debugfs contraption once, and
given I couldn't reproduce the problem I suppose that happens if the
filesystem of the builder runs out of space.

At least in this instance fsck could detect it, so let's add it as a
sanity check.
2018-05-05 19:30:54 +03:00
aszlig
f148c5c4a1
nixos/tests: Fix QEMU flags for SCSI disks
The ability to specify "-drive if=scsi" has been removed in QEMU version
2.12 (introduced in 3e3b39f173).

Quote from https://wiki.qemu.org/ChangeLog/2.12#Incompatible_changes:

> The deprecated way of configuring SCSI devices with "-drive if=scsi"
> on x86 has been removed. Use an appropriate SCSI controller together
> "-device scsi-hd" or "-device scsi-cd" and a corresponding "-blockdev"
> parameter instead.

So whenever the diskInterface is "scsi" we use the new way to specify
the drive and fall back to the deprecated way for the time being. The
reason why I'm not using the new way for "virtio" and "ide" as well is
because there is no simple generic way anymore to specify these.

This also turns the type of the virtualisation.qemu.diskInterface option
to be an enum, so the user knows which values are allowed but we can
also make sure the right value is provided to prevent typos.

I've tested this against a few non-disk-related NixOS VM tests but also
the installer.grub1 test (because it uses "ide" as its drive interface),
the installer.simple test (just to be sure it still works with
"virtio") and all the tests in nixos/tests/boot.nix.

In order to be able to run the grub1 test I had to go back to
8b1cf100cd (which is a known commit where
that test still works) and apply the QEMU update and this very commit,
because right now the test is broken.

Apart from the tests here in nixpkgs, I also ran another[1] test in
another repository which uses the "scsi" disk interface as well (in
comparison to most of the installer tests, this one actually failed
prior to this commit).

All of them now succeed.

[1]: 9b5a119972/tests/system/kernel/bfq.nix

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edostra, @grahamc, @dezgeg, @abbradar, @ts468
2018-04-30 03:02:59 +02:00
John Ericson
ba52ae5048 treewide: isArm -> isAarch32
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.

The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:

```
ISA:             ARMv8   {-A, -R, -M}
                 /    \
Mode:     Aarch32     Aarch64
             |         /   \
Encoding:   A64      A32   T32
```

At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.

The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.

[1]: https://developer.arm.com/products/architecture/a-profile
2018-04-25 15:28:55 -04:00
John Ericson
53686e8995
Merge pull request #38485 from obsidiansystems/nixos-nixpkgs-options
nixpkgs module: Clean up platform options
2018-04-19 14:59:58 -04:00
Robert Schütz
80fc5f2a24 Merge branch 'master' into staging 2018-04-10 09:13:36 +02:00
Tuomas Tynkkynen
487be791d7 nixos/make-ext4-fs: Use closureInfo
exportReferencesGraph is deprecated and doesn't have the generated
initial Nix database contain the SHA256 of the contents of the store
paths, which breaks various things under Nix 2.0.
2018-04-10 00:31:36 +03:00
John Ericson
c6f7d43678 nixpkgs module: Clean up platform options
- `localSystem` is added, it strictly supercedes system

 - `crossSystem`'s description mentions `localSystem` (and vice versa).

 - No more weird special casing I don't even understand

TEMP
2018-04-06 12:41:44 -04:00
Frederik Rietdijk
23741692fa Merge master into staging 2018-04-04 19:36:39 +02:00
Will Dietz
0086036141 Machine.pm: tweak to look for graphical target in waitForX
Current message may not be printed
(nothing in systemd contains that string,
 it comes from elsewhere)
2018-04-03 13:59:16 -05:00
Michael Bishop
3c9e579d1e
make-system-tarball: allow alternate compression methods 2018-04-03 11:30:43 -03:00
Will Dietz
ddd10e0836 nixos testing: significantly reduce deps needed for OCR tests
Instead of imagemagick built with many libraries
(notably librsvg which these days requires rust)
use imagemagick_light with support for libtiff added.
2018-03-31 00:24:15 -05:00
adisbladis
c4c1d979e9
Merge pull request #35844 from adisbladis/jquery-mv
jquery & jquery-ui: Move to nixos/lib/testing/
2018-03-06 15:34:34 +08:00
Eelco Dolstra
cc2eeef4ab
Fix installing the Nixpkgs channel on the installation media
And test that it got installed correctly.
2018-02-27 19:58:23 +01:00
adisbladis
b7ec621e91
jquery & jquery-ui: Move to nixos/lib/testing/
These (outdated) derivations are only used by nixos/lib/testing.nix.

If we want to provide jquery & jquery-ui packages this is better done
in nodePackages.
2018-02-27 20:07:35 +08:00
Eelco Dolstra
d12c9911df
Merge remote-tracking branch 'origin/master' into nix-2.0 2018-02-22 17:28:51 +01:00
Michael Fellinger
8c9a8d6805 Document the makeTest length restriction 2018-02-20 16:36:16 +02:00
Michael Fellinger
0d988d6735 Only allow test names of up to 50 characters 2018-02-20 16:36:16 +02:00
Michael Raskin
1e23a5a6e9
Merge pull request #34795 from oxij/os/nixosLabel-pt2
nixos: labels part 2
2018-02-19 14:45:17 +00:00
Jan Malakhovski
2e6b796761 nixos: rename config.system.nixos* -> config.system.nixos.* 2018-02-18 12:56:30 +00:00
rnhmjoj
c1bed05e34
nixos/networking-interfaces: rename IP addresses/routes options 2018-02-17 14:01:24 +01:00
Franz Pletz
54c7ca34c4
nixos/testing: use the same qemu in the test driver
The qemu_test package is also used for running a NixOS qemu VM.
2018-02-11 23:40:54 +01:00