Commit graph

276 commits

Author SHA1 Message Date
aszlig a5bc11f9eb
nixos/vm-tests: Remove msize mount option
This seems to be the root cause of the random page allocation failures
and @wizeman did a very good job on not only finding the root problem
but also giving a detailed explanation of it in #10828.

Here is an excerpt:

  The problem here is that the kernel is trying to allocate a contiguous
  section of 2^7=128 pages, which is 512 KB. This is way too much:
  kernel pages tend to get fragmented over time and kernel developers
  often go to great lengths to try allocating at most only 1 contiguous
  page at a time whenever they can.

  From the error message, it looks like the culprit is unionfs, but this
  is misleading: unionfs is the name of the userspace process that was
  running when the system ran out of memory, but it wasn't unionfs who
  was allocating the memory: it was the kernel; specifically it was the
  v9fs_dir_readdir_dotl() function, which is the code for handling the
  readdir() function in the 9p filesystem (the filesystem that is used
  to share a directory structure between a qemu host and its VM).

  If you look at the code, here's what it's doing at the moment it tries
  to allocate memory:

    buflen = fid->clnt->msize - P9_IOHDRSZ;

    rdir = v9fs_alloc_rdir_buf(file, buflen);

  If you look into v9fs_alloc_rdir_buf(), you will see that it will try
  to allocate a contiguous buffer of memory (using kzalloc(), which is a
  wrapper around kmalloc()) of size buflen + 8 bytes or so.

  So in reality, this code actually allocates a buffer of size
  proportional to fid->clnt->msize. What is this msize? If you follow
  the definition of the structures, you will see that it's the
  negotiated buffer transfer size between 9p client and 9p server. On
  the client side, it can be controlled with the msize mount option.

  What this all means is that, the reason for running out of memory is
  that the code (which we can't easily change) tries to allocate a
  contiguous buffer of size more or less equal to "negotiated 9p
  protocol buffer size", which seems to be way too big (in our NixOS
  tests, at least).

After that initial finding, @lethalman tested the gnome3 gdm test
without setting the msize parameter at all and it seems to have resolved
the problem.

The reason why I'm committing this without testing against all of the
NixOS VM test is basically that I think we can only go better but not
worse than the current state.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-12-14 17:26:24 +01:00
Eelco Dolstra 75e41b0210 Add Fedora 23 2015-11-25 16:18:15 +01:00
Eelco Dolstra e4eee41ad0 Add Ubuntu 15.10 2015-11-25 15:40:08 +01:00
aszlig 82d88b5bcb
vmTools.commonDebPackages: Add "mawk".
While debugging an issue with running NixOps tests, I found out that the
output from debClosureGenerator is not deterministic.

The reason behind this is the way how Provides and Replaces fields are
handled. I haven't yet found out what's the exact issue, but so far
packages "Provides" are more or less picked at random.

So, running the NixOps Hetzner tests we get either mawk, original-awk or
gawk altering on every invocation.

While for the test it isn't poisionous whether wi have mawk or gawk,
having original-awk certainly is, because live-build only works with
mawk or gawk.

The best solution would obviously be to make debClosureGenerator
deterministic, but in the case of "Provides: awk", we can safely pick
mawk by default, because the latter has a "Priority: required" in its
package description.

This also has the advantage that we can safely cherry-pick this to
release-15.09 because it's very unlikely that we'll break the
debClosureGenerator by adding a dependency to commonDebPackages.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-10-26 20:56:42 +01:00
Eelco Dolstra 7260c04266 debian: Update to 7.9, 8.2 2015-10-08 11:20:26 +02:00
Eelco Dolstra 106738b196 Give more memory for the disk image builder
http://hydra.nixos.org/build/26480662
2015-10-06 15:03:26 +02:00
Eelco Dolstra 69829dd4ff Debian: Update to 8.1 2015-06-12 11:22:59 +02:00
Eelco Dolstra 99d2e80d97 In VM builds, connect stdin to /dev/null
This prevents programs from hanging while trying to read from stdin.
2015-06-04 20:08:56 +02:00
Benjamin Staffin 1db5bafc94 Typo fix! s/wheezy/jessie/ 2015-05-29 11:16:33 +02:00
Benjamin Staffin c7ac0ed3eb Add Debian 8.0
Had to add file type logic in debClosureGenerator because Packages.bz2
is not available for Jessie - only Packages.xz is.
2015-05-29 10:49:20 +02:00
Eelco Dolstra 178c52e7ad Add Ubuntu 15.04 2015-05-22 13:21:02 +02:00
Eelco Dolstra fef86de618 Make module loading work in VMs
Also, execute stage 2 as PID 1.
2015-05-13 17:36:58 +02:00
Eelco Dolstra b97362989f Provide /run in VMs 2015-05-13 17:36:29 +02:00
Eelco Dolstra c3f6c8c582 Mount /dev/shm in VMs 2015-05-07 15:31:36 +02:00
Eelco Dolstra 3a31043b43 fillDiskWithRPMs: Pass --extract-over-symlinks to cpio
http://hydra.nixos.org/build/20003320
2015-02-27 14:00:11 +01:00
Eelco Dolstra fa47d347a4 Fix conflict between fedora-release and generic-release
http://hydra.nixos.org/build/19594340
2015-02-12 17:55:29 +01:00
Eelco Dolstra ff37dc6fb6 Remove tabs 2015-02-12 17:55:29 +01:00
Eelco Dolstra 28ea9050fa Remove ancient RedHat 9 / SuSE 9 images
These are 12 years old and I don't think anybody is using them.
2015-02-12 17:55:29 +01:00
Eelco Dolstra da73688e3d fillDiskWithRPMs: Shut up some warnings 2015-02-12 17:55:28 +01:00
Eelco Dolstra 3786f08d25 Add Fedora 21 2015-02-10 11:32:43 +01:00
Eelco Dolstra 1b1e015e6e Update Debian to 7.8 2015-01-15 14:51:55 +01:00
Eelco Dolstra 7358b8dfda Fix a warning in initrd generation
${pkgs.busybox}/bin and ${pkgs.busybox}/sbin are now the same
directory, so don't copy both.
2015-01-12 17:52:30 +01:00
Eelco Dolstra 872f8c0431 Remove RHEL 7 RC
It no longer exists on the RedHat FTP site.
2015-01-12 15:13:21 +01:00
Eelco Dolstra 02df7cdaa0 Fix CentOS 6.5 URLs 2015-01-12 15:13:05 +01:00
Eelco Dolstra b8e33a5dcb Ubuntu 14.10: Update hashes 2014-12-05 16:02:59 +01:00
Eelco Dolstra 99a2afd784 Update to Debian 7.7 2014-11-20 11:11:21 +01:00
Eelco Dolstra e5513ac4dc Add Ubuntu 14.10 2014-11-20 11:11:08 +01:00
Eelco Dolstra fda5c17a6f vmTools: Fix permissions on /tmp and /dev/* 2014-11-18 14:49:39 +01:00
Shea Levy 3fe41ca763 Debain 7.6-amd64 tarball was updated in place 2014-10-20 15:26:51 -04:00
Shea Levy 652030e85d debian packages tarball was updated in place 2014-10-20 15:23:44 -04:00
Eelco Dolstra 91fa55e702 Remove references to <nixpkgs>
Nixpkgs should not use <nixpkgs> to refer to itself, because $NIX_PATH
might not be set or not point to the same Nixpkgs. It's also
unnecessary.
2014-09-23 15:48:15 +02:00
Mathijs Kwik e80cbedbae VM builds: Use default kernel
(this was set to 3.10 when 3.4 was our default)
See commit d846e97656.
2014-09-23 09:17:01 +02:00
Eelco Dolstra 7e744041f3 rpm-closure.pl: Reduce verbosity 2014-09-05 12:37:07 +02:00
Eelco Dolstra 05e81e0d9f debian: Update to 6.0.10 2014-08-10 15:17:00 +02:00
Eelco Dolstra ff97b7dbe6 Merge remote-tracking branch 'origin/master' into staging 2014-07-14 17:04:55 +02:00
Eelco Dolstra 1245ca3ff7 debian: Update to 7.6 2014-07-14 14:35:09 +02:00
Eelco Dolstra 40f7b0f9df Another attempt to eradicate ensureDir
See c556a6ea46.
2014-06-30 14:56:10 +02:00
Eelco Dolstra 139608dd34 Update Ubuntu Packages.bz2 hashes 2014-05-20 13:39:06 +02:00
aszlig 97dc8a88e5
vm/windows: Wait for migration to finish.
This ensures that the intermediate machine is shut down only after the
migration has finished writing the memory dump to disk, to ensure we
don't end up with empty state files depending on how fast the migration
finished before we actually shut down the VM.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-05-07 07:21:10 +02:00
aszlig bd78e674c5
vm/windows: Exit if VM has dropped out.
This ensures that the builder isn't waiting forever if the Windows VM
drops dead while we're waiting for the controller VM to signal that a
particular command has been executed on the Windows VM. It won't ever
happen in such cases so it doesn't make sense to wait for the timeout.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-05-07 07:21:10 +02:00
Eelco Dolstra 0ff68e81b9 debian: Update to 7.5
[Note from Austin: I think @edolstra forgot to merge this to master.]

(cherry picked from commit 02b056c5b180b4b8ba22ddc3061d78258e2ef98f on
release-14.04)
2014-05-03 10:55:26 -05:00
Austin Seipp 2be1b4c034 build-support/vm: add Ubuntu 14.04 Trusty Tahir
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2014-05-03 10:51:32 -05:00
Rob Vermaas eaedf303a0 Add RHEL7 to vm functions. 2014-05-01 15:13:35 +02:00
Eelco Dolstra 1bae93cb82 debian: Update to 6.0.9 2014-04-25 10:43:26 +02:00
Eelco Dolstra f73ff0529e runInLinuxVM: Set the clock properly 2014-03-18 15:02:09 +01:00
aszlig bd2d120c08
vm/windows: Allow to specify a different arch.
So far, we determined this based on stdenv.is64bit, but there are cases
where you want to run/build a 32bit program on a 64 bit Windows.

This is now possible, by passing windowsImage.arch = "i686" | "x86_64"
to runInWindowsVM. Based an what was passed, the corresponding Cygwin
packages and setup.exe are bootstrapped.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-04 09:22:18 +01:00
aszlig e86ffa08b7
vm/windows: Update SHAs for Cygwin setup.ini.
Another very annoying part. Unfortunately, the only option we might have
here is to include it in nixpkgs or maybe make a fixed Hash on the
result of the closure fetcher.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 23:37:41 +01:00
aszlig 5bd52771ba
vm/windows: Replace binary with cross cygwinSetup.
As the official Cygwin setup binary download doesn't come in snapshots
or even versioned, the fetchurl of setup.exe will frequently fail, which
in turn will annoy us as hell (or at least me).

One warning though: The fetchurl is currently broken and the cross-build
might not work yet for example on mingw32 (mingw-w64 branch on its way),
but the upstream URL has already changed and the new version contains a
bug (not yet tracked down) which breaks our Windows bootstrap process.

So to conclude: If it's already broken, make it at least "less broken".

"Not broken" is coming soon with the merge of the mingw-w64 branch.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 23:21:34 +01:00
aszlig 40635ed343
vm/windows: Remove stray "import <nixpkgs>".
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 22:45:14 +01:00
aszlig ca8cdcda06
vm/windows: Fix typo in runInWindowsVM.
Coincidentally, even with this typo, most tests work anyway, so I didn't
notice it in the first place.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 22:42:10 +01:00