Commit graph

423 commits

Author SHA1 Message Date
Léo Gaspard a5a819e059
Merge pull request #104292 from fgaz/image-contents
nixos/lib/make-disk-image.nix: support content mode and ownership
2020-12-21 19:58:49 +01:00
Francesco Gazzetta 6f21ae7a62 nixos/lib/make-disk-image.nix: support content mode and ownership 2020-12-20 21:22:47 +01:00
Robert Hensing 72a5301610 nixos/lib/testing-python.nix: Wire up passthru 2020-12-09 13:24:03 +01:00
Graham Christensen bc49a0815a
utillinux: rename to util-linux 2020-11-24 12:42:06 -05:00
Frederik Rietdijk f36b838e2a nixos test-driver: fix single line docstrings, fixes #104467
Single line docstrings should have the """ on a single line according to PEP 8.
It seems support for this landed in the latest version of Black.
2020-11-21 09:51:31 +01:00
Emery Hemingway 7e25b71132 nixos: use nativeBuildInputs in make- iso9660-image and system-tarball
The tools used to create iso9660 images and tarballs are independent of
the platform of the closure contained within.
2020-11-18 14:05:30 +01:00
Graham Christensen d78aa080f5
make-disk-image: support legacy+gpt 2020-10-30 15:50:24 -04:00
Andreas Rammhold eba8f5425f
nixos/tests: fix wrong inherit that passes on the nodes attrs
The hydra tarball step would fail due to the nodes attribute not being
properly inherited. Since we can't execute all the tests and release
steps locally anymore (thanks to the JSONification and faster hydra
eval) these errors will probably keep in appearing.

This is hopefully the last of those introduced by me test runner
refactoring.

Error was seen on hydra (https://hydra.nixos.org/build/129282411):
> unpacking sources
> unpacking source archive /nix/store/bp95x52h6nv3j8apxrryyj2rviw682k1-source
> source root is source
> patching sources
> autoconfPhase
> No bootstrap, bootstrap.sh, configure.in or configure.ac. Assuming this is not an GNU Autotools package.
> configuring
> release name is nixpkgs-21.03pre249116.1088f059401
> git-revision is 1088f05940
> building
> no Makefile, doing nothing
> running tests
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> checking Nixpkgs on i686-linux
> checking Nixpkgs on x86_64-linux
> checking Nixpkgs on x86_64-darwin
> checking eval-release.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> checking find-tarballs.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> error: while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:6:1, called from undefined position:
> while evaluating 'operator' at /build/source/maintainers/scripts/find-tarballs.nix:27:16, called from undefined position:
> while evaluating 'immediateDependenciesOf' at /build/source/maintainers/scripts/find-tarballs.nix:39:29, called from /build/source/maintainers/scripts/find-tarballs.nix:27:44:
> while evaluating anonymous function at /build/source/lib/attrsets.nix:234:10, called from undefined position:
> while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:40:37, called from /build/source/lib/attrsets.nix:234:16:
> while evaluating 'derivationsIn' at /build/source/maintainers/scripts/find-tarballs.nix:42:19, called from /build/source/maintainers/scripts/find-tarballs.nix:40:40:
> while evaluating 'canEval' at /build/source/maintainers/scripts/find-tarballs.nix:48:13, called from /build/source/maintainers/scripts/find-tarballs.nix:43:9:
> while evaluating the attribute 'nodes' at /build/source/nixos/lib/testing-python.nix:195:23:
> attribute 'nodes' missing, at /build/source/nixos/lib/testing-python.nix:193:16
> build time elapsed:  0m0.122s 0m0.043s 17m51.526s 0m56.668s
> builder for '/nix/store/96rk3c74vrk6m3snm7n6jhis3j640pn4-nixpkgs-tarball-21.03pre249116.1088f059401.drv' failed with exit code 1
2020-10-27 00:10:31 +01:00
Andreas Rammhold 73635b859d
nixos/tests: fix runInMachine
In 5500dc8 we introduced the --keep-vm-state flag and defaulted to that
flag not being set. This lead to the `runInMachine` tests not longer
working and that going unnoticed for quite some time now.
2020-10-25 20:09:33 +01:00
Andreas Rammhold fa25d84d13
nixos/tests: fix testDriver reference in runInMachine function
In a previous commit I broke this as there is no longer one testDriver
but only a function to generate one based on some QEMU inputs.
2020-10-25 20:09:33 +01:00
Andreas Rammhold 61b09f552c
nixos/tests: format the testing-python.nix file more consistenly 2020-10-25 20:09:33 +01:00
Andreas Rammhold 04100cd281
nixos/tests: restructure test driver so that QEMU is actually overriden
Previously you would be able to override only the QEMU package to be
used in the test runner. Frankly that doesn't help a lot if you are
trying to get a graphical session. The graphical session requires the
option in the NixOS module system to bet set to the correct QEMU
package.

In this commit I moved most of the test node configuration and
transformations into the `mkDriver` function (previously called
`driver`). The motivation was to be able to create a `driver` instance
with a given QEMU package that will be used consistently througout the
test expression.
2020-10-25 20:09:33 +01:00
Konrad Borowski 254d30d4c9 test-driver.py: remove bufsize=1 from Popen calls
According to Python documentation [0], `bufsize=1` is only meaningful in
text mode. As we don't pass in an argument called `universal_newlines`,
`encoding`, `errors` or `text` the file objects aren't opened in text
mode, which means the argument is ignored with a warning in Python 3.8.

    line buffering (buffering=1) isn't supported in binary mode,
    the default buffer size will be used

This commit removes this warning that appared when using
interactive test driver built with `-A driver`. This is done by
removing `bufsize=1` from Popen calls.

The default parameter when unspecified for `bufsize` is `-1` which
according to the documentation will be interpreted as
`io.DEFAULT_BUFFER_SIZE`. As mentioned by a warning, Python already
uses default buffer size when providing `buffering=1` parameter for
file objects not opened in text mode.

[0]: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
2020-10-25 16:22:07 +01:00
rnhmjoj bc2188b083
nixos: fix qemu_test being used in normal VMs
This is an attempt to fixup PR #49403.
2020-10-21 16:38:04 +02:00
Andreas Rammhold 20893b3a70
nixos/tests: expose both the interactive and non-interactive driver
For a lot of the work the non-interactive drivers are enough and it is
probably a good idea to keep it accessible for debugging without
touching the Nix expression.
2020-10-19 17:39:48 +02:00
Andreas Rammhold c096880d46
nixos/tests: make the driver attribute use a rich qemu
Since we previously stripped down the features of `qemu_test` some of
the features users are used to while running tests through the (impure)
driver didn't work anymore. Most notably we lost support for graphical
output and audio. With this change the `driver` attribute uses are more
feature complete version of QEmu compared to the one used in the pure
Nix builds.

This gives us the best of both worlds. Users are able to see the
graphical windows of VMs while CI and regular nix builds do not have to
download all the (unnecessary) dependencies.
2020-10-19 17:39:48 +02:00
zowoq 008de9ca3c nixos/{containers,cri-o,podman}: move copyFile to nixos/lib/utils 2020-09-24 10:01:47 +10:00
Janne Heß 9f33ab62d9
nixos/testing: Add support for specialArgs
Since using flakes disallows the usage of <unstable> (which I use in
some tests), this adds an alternative. By setting specialArgs, all VMs
can get the `unstable` flake input as an arg. This is not possible with
extraConfigurations, as that would lead to infinite recursions.
2020-09-14 00:10:21 +02:00
Félix Baylac-Jacqué e571b315e2
test-driver.py: defaulting keepVmState in Machine init
ecb73fd555 introduced a new keepVmState
CLI flag for test-driver.py. This CLI flags gets forwarded to the
Machine class through create_machine.

It created a regression for the boot tests where __main__ end up not
being evaluated. See
https://github.com/NixOS/nixpkgs/pull/97346#issuecomment-690951837 for
bug report.

Defaulting keepVmState to false when __main__ ends up not being
evaluated.
2020-09-11 10:47:03 +02:00
Félix Baylac-Jacqué ecb73fd555
test-driver.py: fix VM state directory deletion
The previous version of the code would only kick in if the state
directory path pointed at a *file*, which never occurs. Making that
codepath actually work reveals an ordering bug, which this patch fixes
as well.

It also replaces the confusing, imperative case log message "delete VM
state directory" with "deleting VM state directory".

Finally, we hint the user about how to prevent this deletion. IE. by
passing the --keep-vm-state flag.

Bug report:
https://github.com/NixOS/nixpkgs/pull/91046#issuecomment-685568750

Credit goes to Edef for the rebase on top of a recent nixpkgs commit
and for writing most of this commit message.

Co-authored-by: edef <edef@edef.eu>
2020-09-07 12:26:40 +02:00
Florian Klink 98d6b55fdc nixos/testing: remove remaining coverage-data logic
This isn't used anymore as per
https://github.com/NixOS/nixpkgs/pull/72354#discussion_r451031449.
2020-09-05 16:07:59 +02:00
WORLDofPEACE 18348c7829
Merge pull request #96042 from rnhmjoj/loaOf
treewide: completely remove types.loaOf
2020-09-02 08:45:37 -04:00
rnhmjoj 3f8a3246f4
nixos/lib/make-options-doc: remove loaOf subs
Remove the substitution for the <name?> placeholder used by loaOf,
now that the type has been deprecated.
2020-09-02 00:42:51 +02:00
Anders Kaseorg 59b6664f15 Revert "Merge pull request #96254 from Mic92/logging"
This reverts commit 4fc708567f, reversing
changes made to 0e54f3a6d8.

Fixes #96699.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30 15:46:39 -07:00
Anders Kaseorg a0a421bf5e Revert "Merge pull request #96152 from JJJollyjim/colour-test-machines-staging"
This reverts commit 1bff6fe17c, reversing
changes made to 2995fa48cb.

There’s presumably nothing wrong with this PR, except that it
conflicts with reverting #96254 which broke several tests (#96699).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30 15:45:44 -07:00
aszlig b0ac24ae41
nixos/test-driver: Use guest time when using sleep
With the Perl driver, machine.sleep(N) was doing a sleep on the guest
machine instead of the host machine. The new Python test driver however
uses time.sleep(), which instead sleeps on the host.

While this shouldn't make a difference most of the time, it *does*
however make a huge difference if the test machine is loaded and you're
sleeping for a minimum duration of eg. an animation.

I stumbled on this while porting most of all my tests to the new Python
test driver and particularily my video game tests failed on a fairly
loaded machine, whereas they don't with the Perl test driver.

Switching the sleep() method to sleep on the guest instead of the host
fixes this.

Signed-off-by: aszlig <aszlig@nix.build>
2020-08-29 00:55:01 +02:00
Florian Klink 0620184f3f nixos/lib/test*: remove perl test driver
This has been deprecated in 20.03, and all tests have been migrated to
the python framework, effectively making this dead code.
2020-08-27 19:45:38 +02:00
Jamie McClymont d7875caf76 nixos/test: colour machine names 2020-08-27 23:29:55 +12:00
Jörg Thalheim 4fc708567f
Merge pull request #96254 from Mic92/logging 2020-08-26 19:45:24 +01:00
Matthew Bauer ca3fa9c32a
Merge pull request #95956 from matthewbauer/qemu-cpu-max
runInLinuxVM, test-driver: use -cpu max instead of -cpu host
2020-08-26 12:59:57 -05:00
Jörg Thalheim 87214dbd10
nixos/test-driver: re-introduce log()
Appearantly this is used in tests
2020-08-25 14:50:47 +01:00
Jörg Thalheim f3c0a09c76
nixos/testdriver: sort imports 2020-08-25 10:15:24 +01:00
Jörg Thalheim 392415c285
nixos/test-driver: switch to pythons' logging lib
- Less code
- more thread-safe according to @flokli
2020-08-25 10:13:27 +01:00
Jörg Thalheim c1667f85bb
nixos/test-driver: introduce main method
This way we not accidentally use introduce/use global variables.
Also it explictly mark the code for the mypy type checker.
2020-08-25 09:36:37 +01:00
Silvan Mosberger b77d8ead28
Merge pull request #51850 from roberth/nixos-pure
nixos/lib/eval-config.nix: Add extraModules parameter for opt-in purity
2020-08-24 04:29:37 +02:00
Matthew Bauer 47b56e7c19 runInLinuxVM, test-driver: use -cpu max instead of -cpu host
This appears to avoid requiring KVM when it’s not available. This is
what I originally though -cpu host did. Unfortunately not much
documentation available from the QEMU side on this, but this appears
to square with help:

$ qemu-system-x86 -cpu help
...
x86 host                  KVM processor with all supported host features
x86 max                   Enables all features supported by the accelerator in the current host
...

Whether we actually want to support this not clear, since this only
happens when your CPU doesn’t have full KVM support. Some Nix builders
are lying about kvm support though. Things aren’t too slow without it
though.

Fixes https://github.com/NixOS/nixpkgs/issues/85394

Alternative to https://github.com/NixOS/nixpkgs/pull/83920
2020-08-21 23:42:07 -05:00
Janne Heß ff03800d3b
nixos/testing: Fix fail() function
The docs say this behaves as succeed(), but it does not return stdout as
succeed() does. This fixes that behaviour
2020-08-21 21:28:24 +02:00
Jacek Galowicz 2a288cb1da
Merge pull request #93824 from blitz/fix-rpi4-installer
Fix Raspberry Pi 4B SD-Card Install Image
2020-08-08 13:45:02 +02:00
Maximilian Bosch 2ae62ef72f
Merge pull request #94611 from Ma27/rel-pkgs-error-msg
nixos/manual: improve error message for invalid values in `relatedPackages'
2020-08-03 20:17:46 +02:00
Maximilian Bosch 6d19c04416
nixos/manual: improve error message for invalid values in `relatedPackages'
As reported in NixOS discourse[1], tracking down invalid values in
`relatedPackages'[2] (i.e. list-items that don't exist in `pkgs`) is
fairly hard as the message "Invalid package attribute path `foobar'"
is quite unhelpful and the trace doesn't point to the source of the
problem either.

This patch improves the error message by mentioning that the issue is an
invalid `relatedPackages`-declaration in $optionName.

[1] https://discourse.nixos.org/t/invalid-package-attribute-path-nextcloud19/8403/9
[2] https://nixos.org/nixpkgs/manual/#sec-functions-library-options
2020-08-03 16:37:59 +02:00
zowoq 25d7880f17 nixos/lib/*: editorconfig fixes 2020-08-04 00:23:54 +10:00
Emery Hemingway 3f922834b8 nixos/lib/make-disk-image.nix: abritary format input
Pass unrecognized format types as the output file extension to
qemu-img. The motivation is support for "vdi" output.
2020-07-31 15:33:08 +02:00
Keshav Kini 5e86bba082 nixos/boot: some documentation improvements
- Give a more accurate description of how fileSystems.<name/>.neededForBoot
  works

- Give a more detailed description of how fileSystems.<name/>.encrypted.keyFile
  works
2020-07-29 14:39:21 -07:00
Lassulus 77cf1a6581
Merge pull request #89331 from Lassulus/make-disk-image
make-disk-image: add hybrid and dynamic sized images
2020-07-28 20:34:20 +02:00
lassulus 883a6079fd make-disk-image: add hybrid and dynamic sized images 2020-07-27 19:45:55 +02:00
Julian Stecklina fd2047c642 nixos/lib/make-ext4-fs: fix after mkfs.ext4 refactoring
In 9ac1ab10c9 this library function was
refactored to use mkfs.ext4 instead of cptofs. There are two problems:

If populateImageCommands would create no files (which is fine), a cp
invocation would fail due to missing source arguments.

Another problem is that mkfs.ext4 relies on fakeroot to have sane
uid/gids in the generated filesystem image. This currently doesn't
work for cross compiling.
2020-07-26 22:49:55 +02:00
Timo Kaufmann 830a8d6ee1
Merge pull request #74174 from raboof/fix-74047-stable-gpt-disk-guid
make-iso9660-image: produce stable GPT disk GUID
2020-07-21 00:30:57 +02:00
Bas van Dijk 6e7822b8f3 lib: toHex -> toHexString & toBase -> toBaseDigits
This makes the type of these functions more apparent from the name.
2020-07-20 13:14:19 +02:00
Bas van Dijk e15815e885 nixos/tests/networking.nix: test the services.dhcpd4.machines option
This modifies the `router` to not give out a range of IP addresses but
only give out a fixed address based on the MAC address using the
`services.dhcpd4.machines` option.

To get access to the MAC address the `qemuNicMac` function is defined
and exported from `qemu-flags.nix`.
2020-07-20 13:09:27 +02:00
Bas van Dijk 0410f5dff9 nixos/tests: support up to 255 nodes in NixOS tests 2020-07-20 13:09:27 +02:00