Commit graph

116 commits

Author SHA1 Message Date
Marijan Petricevic 5854aaf680 nixos/tests: improve get_unit_info expressiveness on failure 2019-11-08 11:20:54 +01:00
worldofpeace 1e7ddf233a
Merge pull request #72943 from marijanp/port-test-driver-python
Port remaining test-driver functions to python
2019-11-07 20:23:27 +00:00
worldofpeace 6ed6d1af6a
Merge pull request #72947 from tfc/boot-test-port-fix
Boot test port fix
2019-11-07 17:32:43 +00:00
Jacek Galowicz 52ee1026b0 nixos/test: Handle undecodable bytes
This threw exceptions in boot menus
2019-11-07 10:00:39 +01:00
worldofpeace 3780b9e69c
Merge pull request #72835 from tfc/nixos-integration-test-ports
Nixos integration test ports
2019-11-07 01:05:36 +00:00
Marijan 9915a8ca45 nixos/tests: implemented python test-driver version of forwardPort 2019-11-06 22:43:37 +01:00
Marijan 01f79dfc9e nixos/tests: implemented python test-driver version of dumpTTYContents 2019-11-06 22:43:37 +01:00
Jacek Galowicz cf138b4e6b nixos/tests: Fix subprocess launch call for VM monitor
This crashed in the create-script case
2019-11-06 21:48:08 +01:00
Daniel Schaefer afbfef93a7 nixos/tests: Ignore shutdown/crash if not booted
Condition seems to be inverted. Crash and shutdown only make sense, when
the machine is booted; i.e. we return immediately otherwise.
In the Perl test driver this is:

    return unless $self->{booted};
2019-11-06 16:06:43 +01:00
Jacek Galowicz 45b339b736 nixos/tests: Implement python test-driver version of getWindowNames and waitForWindow 2019-11-05 18:07:05 +01:00
Jacek Galowicz 9c54658226 nixos/tests: Implement python test-driver version of requireActiveUnit 2019-11-05 18:07:00 +01:00
Florian Klink ac97edf013 nixos/test: use ptpython as repl 2019-11-04 23:50:27 +01:00
Jacek Galowicz 3a28fefe7d nixos/test: Port test driver to python
Thanks @blitz and @jtraue for help with implementing machine methods
2019-11-04 23:50:27 +01:00
Andrew Childs d2144755a4 nixos-test-driver: allow configuration of net frontend and backend
When IPXE tests were added, an option was added for configuring only
the frontend, and the backend configuration was dropped entirely. This
caused most installer tests to fail.
2019-07-22 13:44:27 +03:00
Nikolay Amiantov 81d35a9d7e nixos-test-driver: support netRomFile
Needed for UEFI PXE netboot testing.
2019-07-15 19:33:26 +03:00
Samuel Dionne-Riel 3aab228d09 Revert "Add ssh backdoor to VM tests infrastructure."
This reverts commit d6e3db44cf.

See #53935 for explanations. In short, it may be causing issues with
tests on the build infrastructure.
2019-01-19 13:24:39 -05:00
Samuel Dionne-Riel 2646a64fbc
Merge pull request #53827 from samueldr/feature/data-in-logs
tests: Logs timing in tests
2019-01-12 12:54:56 -05:00
Samuel Dionne-Riel b28b37eb00 tests: Wait for shell for twice as long (10m)
See #49441 for an earlier attempt, which was subsequently reverted. I am
assuming that doubling the time will be sufficient if the machine is
overloaded since so many of the tests already pass at 5 minutes, while
still not holding back failures for needlessly long.
2019-01-11 22:40:19 -05:00
Samuel Dionne-Riel 5d93e2c01c test-driver: Logs time taken for nests 2019-01-11 22:36:31 -05:00
Samuel Dionne-Riel 1fe0018df8 test-driver: Adds time it took to connect to guest in logs
This will make it possible to track whether the time is generous or not
when ran on hydra.
2019-01-11 22:36:31 -05:00
Sarah Brofeldt 95486ca306 Revert "NixOS tests: Wait for shell for 10x longer (50m)"
This reverts commit 9bc10e1291.
2018-11-05 08:58:08 +00:00
Sarah Brofeldt 9bc10e1291 NixOS tests: Wait for shell for 10x longer (50m) 2018-10-30 09:22:42 +01:00
Tuomas Tynkkynen d6e3db44cf Add ssh backdoor to VM tests infrastructure.
Thanks to @dezgeg for prototype implementation, I've
cleaned it up and added documentation.
2018-09-28 10:53:08 +01:00
aszlig f148c5c4a1
nixos/tests: Fix QEMU flags for SCSI disks
The ability to specify "-drive if=scsi" has been removed in QEMU version
2.12 (introduced in 3e3b39f173).

Quote from https://wiki.qemu.org/ChangeLog/2.12#Incompatible_changes:

> The deprecated way of configuring SCSI devices with "-drive if=scsi"
> on x86 has been removed. Use an appropriate SCSI controller together
> "-device scsi-hd" or "-device scsi-cd" and a corresponding "-blockdev"
> parameter instead.

So whenever the diskInterface is "scsi" we use the new way to specify
the drive and fall back to the deprecated way for the time being. The
reason why I'm not using the new way for "virtio" and "ide" as well is
because there is no simple generic way anymore to specify these.

This also turns the type of the virtualisation.qemu.diskInterface option
to be an enum, so the user knows which values are allowed but we can
also make sure the right value is provided to prevent typos.

I've tested this against a few non-disk-related NixOS VM tests but also
the installer.grub1 test (because it uses "ide" as its drive interface),
the installer.simple test (just to be sure it still works with
"virtio") and all the tests in nixos/tests/boot.nix.

In order to be able to run the grub1 test I had to go back to
8b1cf100cd (which is a known commit where
that test still works) and apply the QEMU update and this very commit,
because right now the test is broken.

Apart from the tests here in nixpkgs, I also ran another[1] test in
another repository which uses the "scsi" disk interface as well (in
comparison to most of the installer tests, this one actually failed
prior to this commit).

All of them now succeed.

[1]: 9b5a119972/tests/system/kernel/bfq.nix

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edostra, @grahamc, @dezgeg, @abbradar, @ts468
2018-04-30 03:02:59 +02:00
Will Dietz 0086036141 Machine.pm: tweak to look for graphical target in waitForX
Current message may not be printed
(nothing in systemd contains that string,
 it comes from elsewhere)
2018-04-03 13:59:16 -05:00
Maximilian Bosch e538e00404
test-driver: support testing user units
It is quite complicated to test services using the test-driver when
declaring user services with `systemd.user.services` such as many
X11-based services like `xautolock.service`.

This change adds an optional `$user` parameter to each systemd-related
function in the test-driver and runs `systemctl --user` commands using
`su -l $user -c ...` and sets the `XDG_RUNTIME_DIR` variable
accordingly and a new function named `systemctl` which is able to run a
systemd command with or without a specified user.

The change can be confirmed with a simple VM declaration like this:

```
import ./nixos/tests/make-test.nix ({ pkgs, lib }:

with lib;

{
  name = "systemd-user-test";

  nodes.machine = {
    imports = [ ./nixos/tests/common/user-account.nix ];

    services.xserver.enable = true;
    services.xserver.displayManager.auto.enable = true;
    services.xserver.displayManager.auto.user = "bob";
    services.xserver.xautolock.enable = true;
  };

  testScript = ''
    $machine->start;
    $machine->waitForX;

    $machine->waitForUnit("xautolock.service", "bob");
    $machine->stopJob("xautolock.service", "bob");
    $machine->startJob("xautolock.service", "bob");
    $machine->systemctl("list-jobs --no-pager", "bob");
    $machine->systemctl("show 'xautolock.service' --no-pager", "bob");
  '';
})
```
2018-01-02 20:14:27 +01:00
Tuomas Tynkkynen 8762490337 nixos/qemu: Add virtio-rng-pci to test-driver
Otherwise AArch64 KVM guests don't have enough entropy at boot.
2017-11-26 11:13:21 +02:00
Joerg Thalheim 00344162a7 Revert "nixos/test-instrumentation: softcode DISPLAY"
This reverts commit 4531ec5955.

This broke waitForX: b11ac5d29d (commitcomment-25486347)

cc @Lassulus
2017-11-08 17:04:39 +00:00
lassulus 4531ec5955 nixos/test-instrumentation: softcode DISPLAY 2017-11-07 23:01:59 +01:00
Bas van Dijk 5b8ff5ed49 graphite: 0.9.15 -> 1.0.2
Fixes: #29961

Also added the option:

  services.graphite.web.extraConfig

for configuring graphite_web.
2017-10-08 03:03:22 +02:00
Graham Christensen 3f40fcabbf
nixos tests: waitForWindow: output a list of windows we see prior to the final check
machine: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d'
    machine: exit status 0
    machine: Last chance to match /(?^:dfiirst configuration)/ on the the window list, which currently contains:
    machine: [i3 con] container around 0xf8a5f0, i3: first configuration, [i3 con] floatingcon around 0xf8c260, [i3 con] container around 0xf8a380, i3bar for output Virtual-1, [i3 con] bottom dockarea Virtual-1, [i3 con] workspace 1, [i3 con] content Virtual-1, [i3 con] top dockarea Virtual-1, [i3 con] output Virtual-1, [i3 con] workspace __i3_scratch, [i3 con] content __i3, [i3 con] pseudo-output __i3, i3
2017-06-20 21:16:35 -04:00
Graham Christensen 1b833015b7
nixos tests: waitForText: output the detected screen content prior to the last attempt
machine: Last chance to match /(?^:BALICE)/ on the screen, which currently contains:
    machine: performing optical character recognition
    machine: sending monitor command: screendump /tmp/nix-build-vm-test-run-sddm.drv-0/ocrin.ppm
    machine: Session Layout

    O O

    0 1 : 0 9

    Wednesday, June 21, 2017

    |_ I

    Select your user and enter password
2017-06-20 21:10:34 -04:00
Graham Christensen 348785eec0
nixos tests: waitUntilTTYMatches: Log TTY contents on last try
If the test has not passed yet, on the last attempt it now outputs:

    machine: Last chance to match /logine: / on TTY2, which currently contains:
    machine: running command: fold -w$(stty -F /dev/tty2 size | awk '{print $2}') /dev/vcs2
    machine: exit status 0
    machine:

    <<< Welcome to NixOS 17.09.git.a804ef4 (x86_64) - tty2 >>>

    machine login:

to help debug the problem. Notice the "logine" typo in my check.
2017-06-20 20:57:39 -04:00
Graham Christensen 56435c1404
nixos tests: retry: Count down to 0, and pass remaining attempts to the sub
Allows test functions to output diagnostic information on failure.
2017-06-20 20:54:33 -04:00
Eelco Dolstra 0996ea8a68
NixOS VM tests: Don't create a setgid group in vde_switch
Nix no longer allows this for security reasons.

http://hydra.nixos.org/build/53993125
2017-06-07 11:57:36 +02:00
Volth e5fbeb2f11 test-driver: sendKeys() support for capital letters 2017-05-19 13:56:24 +00:00
aszlig a443bdc0a6
nixos/testing: Improve quality of OCR
First of all, we're now using ImageMagick to improve the screenshot so
that Tesseract has an esier time to recognize the text. The resulting
image of this post-processing is a scaled up black-and-white version
with the backgrounds almost entirely removed and the text edges a bit
blurred, so the screen shots now more or less resemble an image from a
scanner rather. This is what Tesseract is trained for by default.

As mentioned in the previous commit we now also use Tesseract 4, which
further improves the quality of text recognition.

I've spent countless hours just to test different postprocessing
variants and testing what works best for our tests and this is the one
that worked best so far. It's certainly not perfect and I'd like to
avoid the scaling step but we're way better off than before.

In addition to this, the OCR process is now done without an intermediate
file, solely using pipes.

I've tested this using the following VM tests which have OCR enabled:

 * nixos/tests/chromium.nix -A stable
 * nixos/tests/emacs-daemon.nix
 * nixos/tests/installer.nix -A luksroot
 * nixos/tests/lightdm.nix
 * nixos/tests/plasma5.nix
 * nixos/tests/sddm.nix

All of the tests still succeed and comparing some of the recognition
results to the earlier results it now also detects a lot more text than
before this commit.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-04-11 03:21:53 +02:00
aszlig a5e9668c5c
nixos/test: Fix escaping for copyFileFromHost
A long-time issue and one of the reasons I've never used that function
before. So let's remove that todo-comment and escape the contents
properly.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
2017-02-17 19:45:49 +01:00
Graham Christensen 59e77daf5b
nixos tests: make getTTYText smart about tty size 2017-02-12 18:27:06 -05:00
volth 50ec3fe1ac
test-driver: support punctuation in sendChars 2017-01-08 09:17:05 -05:00
Eelco Dolstra 996e0ef777
test-driver: Show test duration 2016-12-29 16:23:27 +01:00
Graham Christensen cb74fd75d7
login test: Create and use direct reads of the TTY contents. 2016-11-30 00:17:18 -05:00
aszlig cb796ccd09
nixos/test-driver/Logger: Replace invalid UTF-8
Regression introduced by d84741a4bf.

The mentioned commit actually is a good thing, because we now get the
output from the X session.

Unfortunately, for the i3wm test, the i3-config-wizard prints out the
raw keyboard symbols directly coming from xcb, so the output isn't
necessarily proper UTF-8.

As the XML::Writer already expects valid UTF-8 input, we assume that
everything that comes into sanitise() will be UTF-8 from the start. So
we just decode() it using FB_DEFAULT as the check argument so that
every invalid character is replaced by the unicode replacement
character:

https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character

We simply re-oncode it again afterwards and return it, so we should
always get out valid UTF-8 in the log XML.

For more information about FB_DEFAULT and FB_CROAK, have a look at:

http://search.cpan.org/~dankogai/Encode-2.84/Encode.pm#Handling_Malformed_Data

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-05-26 14:24:33 +02:00
Eelco Dolstra ad29b72686 test-driver: Fix "unit X is inactive and there are no pending jobs"
This was causing many random test failures.
2016-05-24 16:33:14 +02:00
Eelco Dolstra 9153d8ed64 Fix X11 tests broken by the removal of -ac
Probably missed a few. Also adding xauth to the system path (it was
already in the closure).
2016-04-12 19:13:47 +02:00
Eelco Dolstra 146c727924 waitForUnit: Fail if the unit reaches inactive state with no pending jobs
This prevents waitForUnit for looping for many minutes before giving
up.
2015-12-30 15:17:52 +01:00
Dan Peebles b6c589b2da Simple EC2 user-data VM test 2015-06-11 22:54:04 -04:00
Eelco Dolstra 0f35f9bb69 Drop obsolete boot=on parameter 2015-06-10 15:28:55 +02:00
aszlig 2475a6a239
nixos/test-driver: Fix bashism in Machine.pm.
Thanks to @bjornfor for the note.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-22 08:10:17 +02:00
aszlig 8be00dc71d
nixos/test-driver: Make tesseract OCR optional.
By default this is now enabled, and it has to be explicitely enabled
using "enableOCR = true". If it is set to false, any usage of
getScreenText or waitForText will fail with an error suggesting to pass
enableOCR.

This should get rid of the rather large dependency on tesseract which
we don't need for most tests.

Note, that I'm using system("type -P") here to check whether tesseract
is in PATH. I know it's a bashism but we already have other bashisms
within the test scripts and we also run it with bash, so IMHO it's not a
problem here.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-22 07:21:58 +02:00
aszlig 8bd025a75e
nixos/test-driver: Add new waitForText function.
As promised in the previous commit, this can be used similarly to
$machine->waitForWindow, where you supply a regular expression and it's
retrying OCR until the regexp matches.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-21 19:55:21 +02:00
aszlig 235c2228ca
nixos/test-driver: Add new getScreenText function.
Basically, this creates a screenshot and throws tesseract at it to
recognize the characters from the screenshot. In order to produce a
result that is well enough, we're using lanczos scaling and scale the
image up to 400% of its original size.

This provides the base functionality for a new Machine method which will
be called waitForText. I originally had that idea long ago when writing
the VM tests for VirtualBox and Chromium, but thought it would be
disproportionate to the case.

The downside however is that VM tests now depend on tesseract, but given
the average runtime of our tests it really shouldn't have a too big
impact and it's only a runtime dependency after all.

Another issue is that the OCR process takes quite some time to finish,
but IMHO it's better (as in more deterministic) than to rely on sleep().

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-21 19:55:21 +02:00
Bob van der Linden c57a912016 nixos: test: add tests for booting installation iso in various ways 2015-03-24 22:09:07 +01:00
Eelco Dolstra 51c349d0cc test-driver: Use netpbm instead of imagemagick
Imagemagick pulls in 100s of megabytes of dependencies.
2014-09-25 00:07:30 +02:00
Bjørn Forsman 8e8f95beda nixos/test-reports: use local copy of jquery
The current way test reports get jquery,

  src="//ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js"

only works when getting reports over http:// or https://, not file://.
Change it so that it works for all protocols by using a local copy of
jquery.

This fixes the issue where locally created and browsed test reports
cannot be navigated properly; clicking the '+' symbol to expand
sub-sections doesn't work.
2014-09-07 15:47:59 +02:00
aszlig 622673495b
nixos/log2html: Remove schema from jQuery URLs.
This allows viewing test logs for example when using a Hydra running
with HTTPS.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-06-27 09:41:11 +02:00
Eelco Dolstra 48601269a9 test-driver: Set context to include machines created by createMachine 2014-05-21 15:55:50 +02:00
Eelco Dolstra bfda72c2f9 Fix waitForX 2014-04-17 17:07:03 +02:00
Eelco Dolstra f0a9703f77 journalctl no longer parses the flag "-bu" 2014-04-17 10:56:10 +02:00
Eelco Dolstra eb22e5f026 Remove ignored argument to sync 2014-04-07 13:22:12 +02:00
Eelco Dolstra de57c0eb66 Shut up Perl warning 2014-03-18 14:10:31 +01:00
Eelco Dolstra 01fc3e5153 Revert "test-driver: Set the date to the current time on boot"
This reverts commit 4e6eae45ee. It
breaks running the test driver interactively (in that it causes all
VMs to be started immediately, which is not always what you wnat).
2014-03-18 14:05:51 +01:00
Shea Levy 4e6eae45ee test-driver: Set the date to the current time on boot
Should allow merging #1816
2014-03-17 22:05:19 -04:00
Eelco Dolstra ad7c518e45 Sync /tmp/xchg to ensure that the coverage data is flushed 2014-03-03 13:57:08 +01:00
Shea Levy 51de280c0a nixos X tests: wait for logind to link a session to the server
There seems to be some race causing failures if an X command gets in before slim starts the session

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-22 14:23:56 -05:00
Eelco Dolstra 5c1f8cbc70 Move all of NixOS to nixos/ in preparation of the repository merge 2013-10-10 13:28:20 +02:00