Commit graph

40 commits

Author SHA1 Message Date
Michael Weiss 4ec2b24603
nixos/tests/chromium: Drop the workaround for Chrome GPU crashes
This regression was fixed by 51d83077ff.
2021-07-25 12:39:45 +02:00
Michael Weiss 7b3c054514
nixos/tests/chromium: Check the version and that it's an official build
This also prints and screenshots the output of chrome://version which
contains useful information.

Outputs (stable, beta, ungoogled, chrome-stable, chrome-beta, chrome-dev):
Chromium	92.0.4515.107 (Official Build) (64-bit)
Chromium        92.0.4515.107 (Official Build) (64-bit)
Chromium        91.0.4472.164 (Official Build, ungoogled-chromium) (64-bit)
Google Chrome   92.0.4515.107 (Official Build) (64-bit)
Google Chrome   92.0.4515.107 (Official Build) beta (64-bit)
Google Chrome   93.0.4577.8 (Official Build) dev (64-bit)
2021-07-25 12:35:21 +02:00
Michael Weiss 11400dcd65
chromium: Check the text rendering
This should catch regressions like #131074 in the future. In that case a
glibc update caused a regression that caused most of the text to become
invisible (just not the "Web Store" we've already been checking for).
2021-07-23 10:15:25 +02:00
Michael Weiss 97570d30c7
chromium: 91.0.4472.164 -> 92.0.4515.107
https://chromereleases.googleblog.com/2021/07/stable-channel-update-for-desktop_20.html

This update includes 35 security fixes.

CVEs:
CVE-2021-30565 CVE-2021-30566 CVE-2021-30567 CVE-2021-30568
CVE-2021-30569 CVE-2021-30571 CVE-2021-30572 CVE-2021-30573
CVE-2021-30574 CVE-2021-30575 CVE-2021-30576 CVE-2021-30577
CVE-2021-30578 CVE-2021-30579 CVE-2021-30580 CVE-2021-30581
CVE-2021-30582 CVE-2021-30583 CVE-2021-30584 CVE-2021-30585
CVE-2021-30586 CVE-2021-30587 CVE-2021-30588 CVE-2021-30589

Note: This won't be the smoothest update. Chromium seems to be fine but
requires gtk3 in $LD_LIBRARY_PATH to find libgtk-3.so.0 (otherwise it
crashes during startup) but Google Chrome fails to initialize
("GPU process exited unexpectedly: exit_code=132") and requires
"--use-gl=angle --use-angle=swiftshader" for hardware(?) acceleration
(which seems to work work fine and performant but SwiftShader should
actually use the CPU instead of the GPU).
2021-07-21 11:20:38 +02:00
Michael Weiss c33015a0c9
nixos/tests/chromium: Print the content of chrome://{sandbox,gpu}
This can be very useful when running the test headless or e.g. when
looking at Hydra logs. Especially the chrome://gpu content contains a
lot of interesting information.
I also decided to refactor the test_new_win() function to avoid
duplicate code and rely less on xdo.
2021-07-10 10:35:29 +02:00
Michael Weiss 8c52061b1f
nixos/tests/chromium: Refactor launching the browser process
It should now be more flexible and less error-prone.
2021-07-10 10:35:27 +02:00
Michael Weiss f9645002a2
nixos/tests/chromium: Fix the test for M92+
Unfortunately there are some regressions in the GPU code that cause
Chromium and Google Chrome to crash, e.g.:
machine # [0709/084047.890436:ERROR:process_memory_range.cc(75)] read out of range[   30.153484] show_signal: 20 callbacks suppressed
machine # [   30.153490] traps: chrome[1036] trap invalid opcode ip:55af03357b29 sp:7ffeaa69ad10 error:0 in chrome[55aefe7a4000+81ec000]
machine #
machine # [0709/084047.955039:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)
machine # [0709/084047.955078:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2)
machine # [   30.126905] systemd[1]: Created slice system-systemd\x2dcoredump.slice.
machine # [   30.137012] systemd[1]: Started Process Core Dump (PID 1038/UID 0).
machine # [   30.571987] systemd-coredump[1039]: Process 1036 (chrome) of user 1000 dumped core.
machine # [992:1021:0709/084048.501937:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=132
machine # [   30.594747] systemd[1]: systemd-coredump@0-1038-0.service: Succeeded.

Hopefully this'll be fixed upstream before the final release (there are
bug reports for it) but for the meantime we have to launch the beta and
dev versions with "--use-gl=angle --use-angle=swiftshader".
2021-07-09 11:30:04 +02:00
Maximilian Bosch 774aba102a
nixosTests.chromium: lint
Note: I didn't execute it entirely because I'd have to build chromium
for this, but the diff appears fine.
2021-05-09 02:28:09 +02:00
Michael Weiss 27955d37e9
nixos/tests/chromium: Simplify the logic (#110715)
- Improve the documentation (Python docstrings for functions and more
  meaningful xdotool script names).
- Make more use of the existing methods (e.g. wait_until_succeeds(), and
  send_key())
  - Note: This increases timeouts from 60 seconds to (currently)
    15 minutes but the defaults from test-driver.py should be fine.

This should make it simpler to read the code, understand the test
output, and drop the custom xdotool scripts in the future.
2021-01-24 20:35:29 +01:00
Michael Weiss 86897c8f63
nixos/tests/chromium: Fix another potential race condition
The create_new_win() function could open multiple windows when used
incorrectly. This change makes sure that a new window will only be
created if the main window could be selected successfully.

This also ignores the out return values as they're never used.
2021-01-23 12:15:56 +01:00
Michael Weiss ff8866e243
nixos/tests/chromium: Add ungoogled-chromium and Google Chrome
By default only `chromium` will be tested but other "channels" can be
selected using e.g.:
nix-build nixos/tests/chromium.nix -A ungoogled

This also adds me as secondary maintainer (I'd like to get notified on
PRs/issues and can review them).
2021-01-20 19:20:53 +01:00
Michael Weiss 8a7a8442c1
nixos/tests/chromium: Fix a race condition
Only execute Ctrl+w to close the currently active window if the
new/secondary window (title: "New Tab") could be selected. This fixes a
test failure since the update to Chromium M88 (cc PR #110010).
Without this additional check the main window (title: "startup done")
could still be selected (and thus will be closed) and the script would
close both windows (i.e. terminate Chromium completely).
2021-01-20 16:57:46 +01:00
Pavol Rusnak 7b0167204d treewide: use https for nixos.org and hydra.nixos.org
tarballs.nixos.org is omitted from the change because urls from there
are always hashed and checked
2020-05-03 22:14:21 -07:00
Jacek Galowicz 2e7d406b11 nixosTests.chromium: Port to Python 2020-02-06 15:51:38 +01:00
worldofpeace c95612a5a2 nixos/display-managers/auto: remove
This module allows root autoLogin, so we would break that for users, but
they shouldn't be using it anyways. This gives the impression like auto
is some special display manager, when it's just lightdm and special pam
rules to allow root autoLogin. It was created for NixOS's testing
so I believe this is where it belongs.
2020-01-29 19:05:46 -05:00
Rickard Nilsson 6edbe844b2 nixos/tests/chromium: Fix nixos-hex.svg hash 2020-01-07 19:42:52 +01:00
Léo Gaspard 6c68fbd4e1
tests: refactor to carry the package set as an argument
This way, the package set will be possible to pass without re-importing
all the time
2018-11-11 23:11:45 +09:00
aszlig 73cdd5a476
nixos/tests/chromium: Fix sandbox info matching
As reported by @andir, the regular expressions that match the sandbox
output are no longer matching in the recent Chromium bump as of
bb03fbc2c8.

Instead of a boolean field that determines whether namespace sandboxes
are on, the namespace sandbox is now an enum within "Layer 1 Sandbox".

I've modified the regular expressions accordingly and also ran the test
for the stable branch, which now succeeds.

Signed-off-by: aszlig <aszlig@nix.build>
Issue: https://github.com/NixOS/nixpkgs/issues/49442
Cc: @bendlas, @andir
2018-11-02 10:23:04 +01:00
Vladimír Čunát cc41aefe44
chromium tests: inherit timeout from the package
/cc #49442.  It should decrease the waste of resources due to abortions.
2018-11-01 20:15:27 +01:00
aszlig 1b1b76f70a
nixos/tests/chromium: Wait 10s after new window
This is a very very very ugly workaround and it's because Chromium seems
to eat keystroke for a few seconds after a new window is created.

I haven't found a better solution yet, so let's at least unbreak the
test until we come up with a better way.

Thanks to @vcunat for bringing this to my attention and also doing the
initial bisect.

The change that brought up this problem was 2b29e40153,
which updated Chromium from version 65.0.3325.181 to version
66.0.3359.117. Unfortunately the upstream changelog[1] is way too large
to actually guess what the breaking change is.

[1]: https://chromium.googlesource.com/chromium/src/+log/65.0.3325.181..66.0.3359.117?pretty=fuller&n=10000

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @bendlas, @vcunat
2018-04-28 17:35:13 +02:00
Michael Raskin c78c764547 nixos/tests.chromium: actually notice the tab crash
Argh, debugging NixOS tests takes forever…

(cherry picked from commit 1afb6e790a6a28a4fc5f310a2b0e5e012eb416fd)
2018-04-09 13:21:53 +02:00
aszlig cd10e3c4ff
nixos/tests/chromium: Run tests as normal user
The tests have failed because Chromium has started up displaying the
following error message in a dialog window:

  Chromium can not be run as root.

  Please start Chromium as a normal user. If you need to run as root for
  development, rerun with the --no-sandbox flag.

So let's run as user "alice" and pass all commands using the small
helper function "ru" (to keep it short, it's for "Run as User").

Tested it by running the "stable" test on x86_64-linux.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Reported-by: @globin
2017-02-07 07:36:56 +01:00
aszlig 4e7eb75a79
nixos/tests/chromium: Fix popup detection
Sometimes it happens that the "Type to search or enter a URL to
navigate" popup doesn't show, but all we need to know at this time is
whether Chromium has finished starting up.

So checking for the "startup done" page is a better option here.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-09 09:41:10 +01:00
aszlig e047d79279
nixos/tests/chromium: Propagate "system" to pkgs
Assigning the channelMap by the function attrset argument at the
top-level of the test expression file may reference a different
architecture than we need for the tests.

So if we get the pkgs attribute by auto-calling, this will lead to test
failure because we have a different architecture for the test than for
the browser.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-03-21 03:50:38 +01:00
aszlig 3bd71b135b
nixos/tests/chromium: Allow overriding channel map
This has been the case before e45c211, but it turns out that it's very
useful to override the channel packages so we can run tests with
different Chromium build options.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-03-20 22:20:39 +01:00
aszlig e45c211b50
nixos/tests/chromium: Split up into subtests
This makes it easier to test just a specific channel rather than to
force testing all builds down the users/testers throat. Especially this
makes it easier to test NixOS channel upgrades only against the Chromium
stable channel instead of just removing the beta/dev channels from the
tests entirely (as done in 69ec09f38a).

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-03-01 01:04:51 +01:00
Eelco Dolstra 69ec09f38a Don't make chromium-beta/dev release blockers
Generally we shouldn't ship pre-release versions anyway, and we
certainly don't want them to be release blockers. Also, chromium
builds are just too slow to have them blocking the channel (see
https://github.com/NixOS/nixpkgs/issues/12794).
2016-02-03 11:49:50 +01:00
Pascal Wittmann 763d89f9c5 tests-chromium: fix link to svg file
closes #11208
2015-11-29 22:43:28 +01:00
Gabriel Ebner 50b1ca6dcc xdotool: 2.20110530.1 -> 3.20150503.1 2015-09-12 13:18:31 +02:00
Eelco Dolstra aa3b13549f Grmbl
http://hydra.nixos.org/build/24983009
2015-08-25 11:26:38 +02:00
Eelco Dolstra 779e585549 chromium: Fix test on i686-linux
http://hydra.nixos.org/build/24982682
2015-08-24 11:05:01 +02:00
Eelco Dolstra d1c30dc5aa Give the chromium test more memory
Hopefully fixes random failures like

http://hydra.nixos.org/build/24861987/nixlog/446
2015-08-20 20:34:35 +02:00
Joachim Schiele 0731489953 all tests: added meta.maintainers section 2015-07-12 12:29:51 +02:00
aszlig 9bc2f77daa
nixos/tests/chromium: Improve sandbox checking.
We no longer need have "SUID sandbox" enabled in the chrome://sandbox
status page and we now also check for "You are adequately sandboxed." to
be absolutely sure that we're running with proper sandboxing.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-07-04 02:31:45 +02:00
aszlig 5f50f23013
nixos/tests/chromium: Detect popup using OCR.
This will make the test a lot more reliable, because we no longer need
to press ESC multiple times hoping that it will close the popup.

Unfortunately in order to run this test I needed to locally revert the
gyp update from a305e6855d.

With the old gyp version however the test runs fine and it's able to
properly detect the popup.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-22 11:23:09 +02:00
aszlig 82bd608de1
tests/chromium: Work around popup close flakiness.
It's not nice to send the escape key over and over again just to ensure
the popup is closed, because even *if* it fails to close the popup 4
times in a row it's just very unlikely that it will be closed. But in
order to make really sure, we might need to do a screenshot and detect
visual changes.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-04-02 14:12:19 +02:00
aszlig 9de4caddc1
nixos/tests/chromium: Check new userns sandbox.
Since Chromium version 42, we have a new user namespaces sandbox in the
upstream project. It's more integrated so the chrome://sandbox page
reports it as "Namespace Sandbox" instead of SUID sandbox, which we were
re-using (or abusing?) in our patch.

So if either "SUID Sandbox" or "Namespace Sandbox" reports with "Yes",
it's fine on our side.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-02-22 07:52:53 +01:00
aszlig eae32fb798
nixos/tests/chromium: Increase VM memory size.
Chromium is quite memory hungry and we frequently get random crashes in
the tests, so let's set it to 1024 MB because new releases of Chromium
most probably won't consume *less* memory.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-01-26 09:31:03 +01:00
aszlig ad87aef2ab
nixos/tests/chromium: Allow to override packages.
Of course, this could be done via packageOverrides, but this is more
explicit and makes it possible to run the tests with various Chromium
overrides.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-11-19 15:54:19 +01:00
aszlig 0e4c1cc066
nixos: Add rudimentary VM tests for Chromium.
Currently, the test is only for testing the user namespace sandbox and
even that isn't very representative, because we're running the tests as
root.

But apart from that, we should have functionality for opening/closing
windows and the main goal here is to get them as deterministic as
possible, because Chromium usually isn't very nice to chained xdotool
keystrokes.

And of course, the most important "test" we have here: We know at least
whether Chromium works _at_all_.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-11-18 18:41:56 +01:00