Commit graph

395 commits

Author SHA1 Message Date
Vladimír Čunát 30f14243c3 Merge branch 'master' into closure-size
Comparison to master evaluations on Hydra:
  - 1255515 for nixos
  - 1255502 for nixpkgs
2016-04-10 11:17:52 +02:00
Vladimír Čunát d1df28f8e5 Merge 'staging' into closure-size
This is mainly to get the update of bootstrap tools.
Otherwise there were mysterious segfaults:
https://github.com/NixOS/nixpkgs/pull/7701#issuecomment-203389817
2016-04-07 14:40:51 +02:00
Vladimír Čunát aa670eb503 vmTools: update debian jessie 8.3 -> 8.4
Their in-place updates break download hashes...
2016-04-05 14:32:04 +02:00
Lluís Batlle i Rossell 635c99ce87 vm: allow overriding QEMU_OPTS / memSize for images.
It's nice to be able to create disk images with -smp 4
in qemu.
2016-04-01 10:32:59 +02:00
Vladimír Čunát ab15a62c68 Merge branch 'master' into closure-size
Beware that stdenv doesn't build. It seems something more will be needed
than just resolution of merge conflicts.
2016-04-01 10:06:01 +02:00
Lluís Batlle i Rossell ab93f8c137 Making vm's qemu cache=unsafe. Faster.
I don't think it's unsafe, if it's meant for nix expressions.
2016-03-31 09:27:25 +02:00
Lluís Batlle i Rossell e21dd19168 Making vm's interactive shell handle the terminal well. 2016-03-31 09:27:14 +02:00
Nicolas B. Pierron 6313a5698a Replace references to all-packages.nix, by references to the top-level of nixpkgs repository. 2016-03-13 18:25:52 +00:00
Vladimír Čunát 09af15654f Merge master into closure-size
The kde-5 stuff still didn't merge well.
I hand-fixed what I saw, but there may be more problems.
2016-03-08 09:58:19 +01:00
Lluís Batlle i Rossell 202ebf794c vm/rpm/rpm-closure.pl: make it deterministic
Some recent perl version introduced "keys" to return the keys
in random order. As some of the packages are solved by "provides" and
based on the order, this randomness affects what packages get into the
closure.

This problem may be in other nix perl scripts.
2016-03-01 11:02:42 +01:00
Cole Mickens 718848d5aa azure: package qemu @ 2.2.0
This commit packages qemu-220. This package is qemu-2.2.0
and is only used with Azure.
2016-02-18 21:08:28 -08:00
Vladimír Čunát d039c87984 Merge branch 'master' into closure-size 2016-02-14 08:33:51 +01:00
Tuomas Tynkkynen 3cc7fa0be9 treewide: Mass replace 'lzma}/bin' to refer the 'bin' output 2016-02-01 20:46:15 +02:00
Eelco Dolstra 9855cbcbf8 debian: 8.2 -> 8.3 2016-01-30 11:08:37 +01:00
Vladimír Čunát f9f6f41bff Merge branch 'master' into closure-size
TODO: there was more significant refactoring of qtbase and plasma 5.5
on master, and I'm deferring pointing to correct outputs to later.
2015-12-31 09:53:02 +01:00
aszlig a5bc11f9eb
nixos/vm-tests: Remove msize mount option
This seems to be the root cause of the random page allocation failures
and @wizeman did a very good job on not only finding the root problem
but also giving a detailed explanation of it in #10828.

Here is an excerpt:

  The problem here is that the kernel is trying to allocate a contiguous
  section of 2^7=128 pages, which is 512 KB. This is way too much:
  kernel pages tend to get fragmented over time and kernel developers
  often go to great lengths to try allocating at most only 1 contiguous
  page at a time whenever they can.

  From the error message, it looks like the culprit is unionfs, but this
  is misleading: unionfs is the name of the userspace process that was
  running when the system ran out of memory, but it wasn't unionfs who
  was allocating the memory: it was the kernel; specifically it was the
  v9fs_dir_readdir_dotl() function, which is the code for handling the
  readdir() function in the 9p filesystem (the filesystem that is used
  to share a directory structure between a qemu host and its VM).

  If you look at the code, here's what it's doing at the moment it tries
  to allocate memory:

    buflen = fid->clnt->msize - P9_IOHDRSZ;

    rdir = v9fs_alloc_rdir_buf(file, buflen);

  If you look into v9fs_alloc_rdir_buf(), you will see that it will try
  to allocate a contiguous buffer of memory (using kzalloc(), which is a
  wrapper around kmalloc()) of size buflen + 8 bytes or so.

  So in reality, this code actually allocates a buffer of size
  proportional to fid->clnt->msize. What is this msize? If you follow
  the definition of the structures, you will see that it's the
  negotiated buffer transfer size between 9p client and 9p server. On
  the client side, it can be controlled with the msize mount option.

  What this all means is that, the reason for running out of memory is
  that the code (which we can't easily change) tries to allocate a
  contiguous buffer of size more or less equal to "negotiated 9p
  protocol buffer size", which seems to be way too big (in our NixOS
  tests, at least).

After that initial finding, @lethalman tested the gnome3 gdm test
without setting the msize parameter at all and it seems to have resolved
the problem.

The reason why I'm committing this without testing against all of the
NixOS VM test is basically that I think we can only go better but not
worse than the current state.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-12-14 17:26:24 +01:00
Luca Bruno a412927924 Merge remote-tracking branch 'origin/master' into closure-size 2015-11-25 21:37:30 +01:00
Eelco Dolstra 75e41b0210 Add Fedora 23 2015-11-25 16:18:15 +01:00
Eelco Dolstra e4eee41ad0 Add Ubuntu 15.10 2015-11-25 15:40:08 +01:00
Vladimír Čunát 333d69a5f0 Merge staging into closure-size
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
2015-11-20 14:32:58 +01:00
aszlig 82d88b5bcb
vmTools.commonDebPackages: Add "mawk".
While debugging an issue with running NixOps tests, I found out that the
output from debClosureGenerator is not deterministic.

The reason behind this is the way how Provides and Replaces fields are
handled. I haven't yet found out what's the exact issue, but so far
packages "Provides" are more or less picked at random.

So, running the NixOps Hetzner tests we get either mawk, original-awk or
gawk altering on every invocation.

While for the test it isn't poisionous whether wi have mawk or gawk,
having original-awk certainly is, because live-build only works with
mawk or gawk.

The best solution would obviously be to make debClosureGenerator
deterministic, but in the case of "Provides: awk", we can safely pick
mawk by default, because the latter has a "Priority: required" in its
package description.

This also has the advantage that we can safely cherry-pick this to
release-15.09 because it's very unlikely that we'll break the
debClosureGenerator by adding a dependency to commonDebPackages.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-10-26 20:56:42 +01:00
Eelco Dolstra 7260c04266 debian: Update to 7.9, 8.2 2015-10-08 11:20:26 +02:00
Eelco Dolstra 106738b196 Give more memory for the disk image builder
http://hydra.nixos.org/build/26480662
2015-10-06 15:03:26 +02:00
Vladimír Čunát 5227fb1dd5 Merge commit staging+systemd into closure-size
Many non-conflict problems weren't (fully) resolved in this commit yet.
2015-10-03 13:33:37 +02:00
Eelco Dolstra 69829dd4ff Debian: Update to 8.1 2015-06-12 11:22:59 +02:00
Eelco Dolstra 99d2e80d97 In VM builds, connect stdin to /dev/null
This prevents programs from hanging while trying to read from stdin.
2015-06-04 20:08:56 +02:00
Benjamin Staffin 1db5bafc94 Typo fix! s/wheezy/jessie/ 2015-05-29 11:16:33 +02:00
Benjamin Staffin c7ac0ed3eb Add Debian 8.0
Had to add file type logic in debClosureGenerator because Packages.bz2
is not available for Jessie - only Packages.xz is.
2015-05-29 10:49:20 +02:00
Eelco Dolstra 178c52e7ad Add Ubuntu 15.04 2015-05-22 13:21:02 +02:00
Eelco Dolstra fef86de618 Make module loading work in VMs
Also, execute stage 2 as PID 1.
2015-05-13 17:36:58 +02:00
Eelco Dolstra b97362989f Provide /run in VMs 2015-05-13 17:36:29 +02:00
Eelco Dolstra c3f6c8c582 Mount /dev/shm in VMs 2015-05-07 15:31:36 +02:00
Vladimír Čunát 3b9ef2c71b fix "libc}/lib" and similar references
Done mostly without any verification.
I didn't bother with libc}/include, as the path is still correct.
2015-05-05 11:52:08 +02:00
Eelco Dolstra 3a31043b43 fillDiskWithRPMs: Pass --extract-over-symlinks to cpio
http://hydra.nixos.org/build/20003320
2015-02-27 14:00:11 +01:00
Eelco Dolstra fa47d347a4 Fix conflict between fedora-release and generic-release
http://hydra.nixos.org/build/19594340
2015-02-12 17:55:29 +01:00
Eelco Dolstra ff37dc6fb6 Remove tabs 2015-02-12 17:55:29 +01:00
Eelco Dolstra 28ea9050fa Remove ancient RedHat 9 / SuSE 9 images
These are 12 years old and I don't think anybody is using them.
2015-02-12 17:55:29 +01:00
Eelco Dolstra da73688e3d fillDiskWithRPMs: Shut up some warnings 2015-02-12 17:55:28 +01:00
Eelco Dolstra 3786f08d25 Add Fedora 21 2015-02-10 11:32:43 +01:00
Eelco Dolstra 1b1e015e6e Update Debian to 7.8 2015-01-15 14:51:55 +01:00
Eelco Dolstra 7358b8dfda Fix a warning in initrd generation
${pkgs.busybox}/bin and ${pkgs.busybox}/sbin are now the same
directory, so don't copy both.
2015-01-12 17:52:30 +01:00
Eelco Dolstra 872f8c0431 Remove RHEL 7 RC
It no longer exists on the RedHat FTP site.
2015-01-12 15:13:21 +01:00
Eelco Dolstra 02df7cdaa0 Fix CentOS 6.5 URLs 2015-01-12 15:13:05 +01:00
Eelco Dolstra b8e33a5dcb Ubuntu 14.10: Update hashes 2014-12-05 16:02:59 +01:00
Eelco Dolstra 99a2afd784 Update to Debian 7.7 2014-11-20 11:11:21 +01:00
Eelco Dolstra e5513ac4dc Add Ubuntu 14.10 2014-11-20 11:11:08 +01:00
Eelco Dolstra fda5c17a6f vmTools: Fix permissions on /tmp and /dev/* 2014-11-18 14:49:39 +01:00
Shea Levy 3fe41ca763 Debain 7.6-amd64 tarball was updated in place 2014-10-20 15:26:51 -04:00
Shea Levy 652030e85d debian packages tarball was updated in place 2014-10-20 15:23:44 -04:00
Eelco Dolstra 91fa55e702 Remove references to <nixpkgs>
Nixpkgs should not use <nixpkgs> to refer to itself, because $NIX_PATH
might not be set or not point to the same Nixpkgs. It's also
unnecessary.
2014-09-23 15:48:15 +02:00
Mathijs Kwik e80cbedbae VM builds: Use default kernel
(this was set to 3.10 when 3.4 was our default)
See commit d846e97656.
2014-09-23 09:17:01 +02:00
Eelco Dolstra 7e744041f3 rpm-closure.pl: Reduce verbosity 2014-09-05 12:37:07 +02:00
Eelco Dolstra 05e81e0d9f debian: Update to 6.0.10 2014-08-10 15:17:00 +02:00
Eelco Dolstra ff97b7dbe6 Merge remote-tracking branch 'origin/master' into staging 2014-07-14 17:04:55 +02:00
Eelco Dolstra 1245ca3ff7 debian: Update to 7.6 2014-07-14 14:35:09 +02:00
Eelco Dolstra 40f7b0f9df Another attempt to eradicate ensureDir
See c556a6ea46.
2014-06-30 14:56:10 +02:00
Eelco Dolstra 139608dd34 Update Ubuntu Packages.bz2 hashes 2014-05-20 13:39:06 +02:00
aszlig 97dc8a88e5
vm/windows: Wait for migration to finish.
This ensures that the intermediate machine is shut down only after the
migration has finished writing the memory dump to disk, to ensure we
don't end up with empty state files depending on how fast the migration
finished before we actually shut down the VM.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-05-07 07:21:10 +02:00
aszlig bd78e674c5
vm/windows: Exit if VM has dropped out.
This ensures that the builder isn't waiting forever if the Windows VM
drops dead while we're waiting for the controller VM to signal that a
particular command has been executed on the Windows VM. It won't ever
happen in such cases so it doesn't make sense to wait for the timeout.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-05-07 07:21:10 +02:00
Eelco Dolstra 0ff68e81b9 debian: Update to 7.5
[Note from Austin: I think @edolstra forgot to merge this to master.]

(cherry picked from commit 02b056c5b180b4b8ba22ddc3061d78258e2ef98f on
release-14.04)
2014-05-03 10:55:26 -05:00
Austin Seipp 2be1b4c034 build-support/vm: add Ubuntu 14.04 Trusty Tahir
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2014-05-03 10:51:32 -05:00
Rob Vermaas eaedf303a0 Add RHEL7 to vm functions. 2014-05-01 15:13:35 +02:00
Eelco Dolstra 1bae93cb82 debian: Update to 6.0.9 2014-04-25 10:43:26 +02:00
Eelco Dolstra f73ff0529e runInLinuxVM: Set the clock properly 2014-03-18 15:02:09 +01:00
aszlig bd2d120c08
vm/windows: Allow to specify a different arch.
So far, we determined this based on stdenv.is64bit, but there are cases
where you want to run/build a 32bit program on a 64 bit Windows.

This is now possible, by passing windowsImage.arch = "i686" | "x86_64"
to runInWindowsVM. Based an what was passed, the corresponding Cygwin
packages and setup.exe are bootstrapped.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-04 09:22:18 +01:00
aszlig e86ffa08b7
vm/windows: Update SHAs for Cygwin setup.ini.
Another very annoying part. Unfortunately, the only option we might have
here is to include it in nixpkgs or maybe make a fixed Hash on the
result of the closure fetcher.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 23:37:41 +01:00
aszlig 5bd52771ba
vm/windows: Replace binary with cross cygwinSetup.
As the official Cygwin setup binary download doesn't come in snapshots
or even versioned, the fetchurl of setup.exe will frequently fail, which
in turn will annoy us as hell (or at least me).

One warning though: The fetchurl is currently broken and the cross-build
might not work yet for example on mingw32 (mingw-w64 branch on its way),
but the upstream URL has already changed and the new version contains a
bug (not yet tracked down) which breaks our Windows bootstrap process.

So to conclude: If it's already broken, make it at least "less broken".

"Not broken" is coming soon with the merge of the mingw-w64 branch.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 23:21:34 +01:00
aszlig 40635ed343
vm/windows: Remove stray "import <nixpkgs>".
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 22:45:14 +01:00
aszlig ca8cdcda06
vm/windows: Fix typo in runInWindowsVM.
Coincidentally, even with this typo, most tests work anyway, so I didn't
notice it in the first place.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-03-03 22:42:10 +01:00
aszlig 88ea17284e
vmTools: Hook in runInWindowsVM function.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 06:21:48 +01:00
aszlig 895a683d39
vm/windows: Add setup.ini for 32bit architecture.
Also update 64bit setup.ini and check whether we have a 64 bit stdenv in
order to choose the proper Cygwin version. Otherwise we now have the
setup.ini for 32bit available as well.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 06:20:24 +01:00
aszlig 623f1940c0
vm/windows: Make list of dependencies explicit.
So far, the VMs have always been using the native architecture, because
it was reimporting <nixpkgs> several times. Now, we propagate a list of
packages down to all sub-imports, which not only makes clearer which
dependencies a part actually has, but also will make it easier in case
we want to refactor those parts to use callPackage.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 05:43:34 +01:00
aszlig d8e66722a3
vm/windows: Factor out bootstrapping process.
This now isolates the vmTools integration from the bootstrap process and
thus removes our fixed Windows ISO and product key. The latter can now
be provided by an attribute "windowsImage" to runInWindowsVM.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:01 +01:00
aszlig e40f41e505
vm/windows: Collect exit code from xchg.
This is the last item that was missing to get a fully working
runInWindowsVM function. Apart from checking exit codes, we also now
have preVM/postVM hooks which we can use to write arbitrary constructs
around this architecture, without the need to worry about specific
details.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:00 +01:00
aszlig b01c9624cf
vm/windows: Add new runInWindowsVM function.
This function is quite similar to runInLinuxVM, but also ensures that
the builder is run decoupled of the Nix store and using the userland
inside the VM.

We're now picking up the environment variables saved in the previous
commit.

The reason we suppress all errors from the source operation is that it
would emit a ton of errors because we're trying to set read-only
variables.

Also, detecting whether the origBuilder is using the default builder
from the stdenv is currently a bit of a workaround until we have a
specialized pseudo-cross-stdenv someday in the future[TM].

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:00 +01:00
aszlig dd8b0fcf52
vm/windows: Save envirenoment before running VM.
Later, when we start the actual builder, we're going to restore those
environment variables. We're using "(set; declare -p)", here, because
the former is just printing _all_ environment variables, even those not
supported, and the latter only lists specifically declared variables,
which also encludes exports.

The "declare -p" command also emits those variables in a format similar
to the "export" command.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:52:00 +01:00
aszlig 707b7ad1bd
vm/windows: Generate mounts from an attribute set.
This is mainly to make it easier to quickly change mappings, without
making room for errors such as typos.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:59 +01:00
aszlig 5258bbe4c9
vm/windows: Create fstab entries in suspended VM.
Cygwin initializes mounts on _every_ login via SSH and doesn't keep them
consistently like on Unix systems, that's why we need to also add fstab
entries for the bind mounts to the store and xchg shares.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:59 +01:00
aszlig d16dae8d32
vm/windows: Use &&, not set -e in runAndSuspend.
We now have proper quoting, so we no longer need the workaround using
"set -e".

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:59 +01:00
aszlig 084238c6ec
vm/windows: Fix perms of the Samba shares.
We now map all guest accounts to the root user, because in the end the
permissions of the current user boil down to the build user of the Nix
builder of the host. That way it's not possible to gain more permissions
at all and just makes the VM communication a lot easier.

However, setting "writable" to yes instead of "read only" to no doesn't
change anything here, I just found it to be clearer.

Also, we now no longer need to have a "nobody" user.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:59 +01:00
aszlig 5241bb1d5e
vm/windows: Avoid shadowing the Nix store.
This is done by putting the non-initrd /nix/store into a subdirectory,
which we then chroot to and pass along the SSH command.

Also, we now collect the exit code after the chroot command and power
off the VM thereafter, because the store is no longer shadowed and we
have still access to the busybox inside the initrd.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:58 +01:00
aszlig a1d7974acd
vm/windows: Remove Samba from installMode.
This should trim down possible dependencies on the base installation and
hereby reduce the need for reinstallation of the damn VM to only changes
that affect the Windows installation and the base Cygwin + OpenSSH
setup.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:58 +01:00
aszlig fedf13e6cf
vm/windows: Implement and use "xchg" share.
This now finally introduces our xchg share and also uses it for
exchanging state while suspending a VM. However, accessing the _real_
Nix store still isn't possible because we're shadowing the directory in
the initrd.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:58 +01:00
aszlig 0ce1fd07fe
vm/windows: Reintroduce setting up /nix/store.
Now we're doing this at the point where we're saving the VM state.
Unfortunately it's not quite right, because the controller VM is shut
down _before_ we're saving the state, so the share gets disconnected
despite autodisconnect being deactivated during setup.

We can get around this issue by finally introducing the xchg share,
which is the last item to be implemented before we can merge to master.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:57 +01:00
aszlig aa65a7057f
vm/windows: Properly escape shell command.
Security-wise it's not a big issue because we're still sandboxed, but I
really don't want to write something like \\\\\\\\192.168.0.2\\\\share
in order to set up network shares.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:57 +01:00
aszlig cfa859d792
vm/windows: Don't init /nix/store on install.
We're going to do this during the suspendedVM phase, so we're able to
more easily change the shares without reinstalling the whole VM.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:57 +01:00
aszlig 3e91192f07
vm/windows: Wait for VDE switch to startup.
This could possibly cause flapping whenever qemu is too fast in starting
up. As we are running with the shell's -e flag, the socat check also
ensures that the VDE switch is properly started and causes the whole
build to fail, should it not start up within 20 seconds.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:56 +01:00
aszlig c731467e2c
vm/windows: Split install into several stages.
These stages are in particular:

 * Install of the bare Windows VM with Cygwin and shut down.
 * Boot up the same VM again without the installation media and dump the
   VMs memory to state.gz.
 * Resume from state.gz and build whatever we want to build.

Every single stage involves a new "controller", which is more like an
abstraction on the Nix side that constructs the madness described in
276b72fb93.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:51:56 +01:00
aszlig 5105e7f0bf
vm/windows: Update sha256 of Cygwin's setup.ini.
This is kinda stupid to do every little time the file is automatically
regenerated upstream. But let's see how often that happens and whether
it will become a major annoyance or not, and if yes, we might be forced
to include it in our source tree.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:50:55 +01:00
aszlig 9b1862ca1f
vm/windows: Move creating SSH key into install/.
This SSH key is specifically only for accessing the installed Cygwin
within the Windows VM, so we only need to expose the private key. Yes,
you heard right, the private key. It's not security-relevant because the
machine is completely read-only, only exposed to the filesystem and
networking is not available.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:50:54 +01:00
aszlig 4e21215d52
vm/windows: Move the installer into install/.
At least the largest portion of the installer, because in the end we
don't want the installer to *actually* save the state but only prepare
the base image.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:50:54 +01:00
aszlig 276b72fb93
vm: Introduce new Windows VM installer for Cygwin.
After quite a lot of fighting against Windows and its various
limitations, this new is the base architecture for installing and
accessing the Windows VM and thus the Cygwin environment inside it:

                .------------.
          .---> | vde_switch |
          |     `-[#]----[#]-'
          |        |      |
         ,'       .'      `---.___
       ,'    192.168.0.1          `.
       |          |            192.168.0.2
      ,'    _____[#]____           |
    ,'     |            |   ______[#]______
    |      | Windows VM |  |    .--'       |
    |      |____________|  |    |          |
    |             |  /|\   |  .-|          |
    | .---------. |   |    |  | |          |
  .-|-| manager |-'   |    |  | |          |
  | | `---------'     |    |  | |          |
  | |                 |    |  | |          |
  | | .-------------. |    | Samba         |
  | | | BOOTSTRAP   | |    |  | |          |
  | | |-------------| |    |  | |   .------|
  | `-| spawn VMs   |-+--> |  | `---| xchg | <-------.
  |   |-------------|      |  | .---^------|         |
  |   | install     |---.  |  `-| nixstore | <----.  |
  |   |-------------|   |  |    `----------|      |  |
  |---| suspend VM  |   |  |               |      |  |
  |   `------.------'   |  | Controller VM |      |  |
  |          |          |  |_______________|      |  |
  |       .--'          |         /|\            VirtIO
  |       |           __|__________:____________  |  |
  |      \|/         |  |          `.           | |  |
  | .------------.   |  |           :           | |  |
  | | REAL BUILD |   |  |   .-------^--------.  | |  |
  | |------------|   |  `-> | serial console |  | |  |
  `-| revive VM  |   |      `----------------'  | |  |
    |------------|   |------------.             | |  |
    | build      |-->| /nix/store >>>-----------|-'  |
    |------------|   |------------|             |    |
    | collect    |<--| xchg       >>>-----------|----'
    `-----.------'   |------------'             |
          |          |                          |
         \|/         |    |  |  __   ___  |     |
                     |    |--| |  | (__  -|-    |
    F I N I S H E D  |    |  | |__| ___)  |     |
                     |__________________________|

This might look a bit overwhelming, but let me try to explain:

We're starting at the base derivation ("BOOTSTRAP" above), where we
actually install the Cygwin envirenment. Over there we basically fire up
a vde_switch process and two virtual machines: One is the Windows
machine, the other is a NixOS machine, which serves as some kind of
proxy between the host and the Windows machine.

The reason we're doing this, is because we don't have a lot of options
for sharing files between a stock Windows machine and the host. In
earlier experiments, I've tried to communicate with the Windows guest by
using pipes and OpenSSH, but obviously this wasn't a big speed rush (or
to say it bluntly: It was fucking slow).

Using TCP/IP directly for accessing the guest would have been another
option, but it could lead to possible errors when the port or a range of
ports are in use at the Host system. Also, we would need to punch a hole
into the sandbox of the Nix builder (as it doesn't allow networking),
which in turn will possibly undermine deterministic builds/runs (well,
at least as deterministic as it can be, we're running Windows,
remember?).

So, let's continue: The responsibility of the NixOS (controller) VM is
to just wait until an SSH port becomes available on the Windows VM,
whereas the Windows VM itself is installed using an unattended
installation file provided via a virtual floppy image.

With the installation of the basic Windows OS, we directly install
Cygwin and start up an OpenSSH service.

At this point the bootstrapping is almost finished and as soon as the
port is available, the controller VM sets up Samba shares and makes it
available as drive letters within Windows and as bind mounts (for
example /nix/store) within Cygwin.

Finally we're making a snapshot of the memory of the Windows VM in order
to revive it within a few seconds when we want to build something.

Now, the build process itself is fairly straightforward: Revive VM and
build based on existing store derivations and collect the result _and_
the exit code from the xchg share/directory.

Conclusion: This architecture may sound a bit complicated, but we're
trying to achieve deterministic and reproducable builds and/or test
runs.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-02-26 04:50:53 +01:00
Petr Rockai 97721af469 VMs: Add Fedora 17 to the list of distros (only had 16 and 18). 2014-02-15 12:57:21 +01:00
Eelco Dolstra 8ba1fdef00 debian: Update to 7.4 2014-02-14 20:29:43 +01:00
Shea Levy b529a34b4a Add Fedora 20
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-19 20:50:46 -05:00
Rob Vermaas 1625743902 Fix Fedora8 vm builds. 2014-01-15 21:32:38 +01:00
Rob Vermaas ed9520bfb5 Allow specifying extra qemu flags using QEMU_OPTS. Replace CentOS 6.3 with CentOS 6.5 (6.3 is removed from site). 2014-01-14 22:51:26 +01:00
Eelco Dolstra 3abcd7e268 debian: Update to 7.3 2014-01-06 15:51:00 +01:00
Eelco Dolstra 4747796a9e Build RPMs for i686 rather than i386
The default target (i386-linux) causes flags like "-march i386" to be
added, which breaks on recent Fedora releases (18 and up), resulting
in errors like:

  /usr/lib/gcc/i686-redhat-linux/4.7.2/../../../../include/c++/4.7.2/ext/atomicity.h:48: undefined reference to `__atomic_fetch_add_4'

So set the target to i686-linux.

http://hydra.nixos.org/build/6567357
2013-10-23 12:55:07 +02:00
Eelco Dolstra 391de89913 Debian 7: Update to 7.2
Also rename "debian70" to "debian7" to reflect the Wheezy's new
versioning scheme.
2013-10-23 11:30:51 +02:00
Eelco Dolstra d6f97c3601 Debian 6: Update to 6.0.8 2013-10-23 11:17:20 +02:00
Eelco Dolstra 27e91e0044 Add Fedora 19 2013-10-23 11:15:45 +02:00
Eelco Dolstra 34fcf33c0b Add Ubuntu 13.10 2013-10-23 11:08:12 +02:00
Eelco Dolstra d846e97656 VM builds: Use Linux 3.10
3.4 apparently gives corrupt 9pfs data on x86.  See e.g.

  http://hydra.nixos.org/build/5661036

where reading /nix/store/kfldnrrsq0lbbv13gjxdfb3vb1sbaz88-vm-run-stage2
gives garbage.  Strangely it doesn't happen here:

  http://hydra.nixos.org/build/5658978

Ah well.
2013-08-07 14:48:22 +02:00
Eelco Dolstra 4342a32deb VM builds: Reduce kernel verbosity 2013-08-01 14:35:31 +02:00
Eelco Dolstra d077851b7d VM builds: Use the default kernel 2013-08-01 14:35:31 +02:00
Eelco Dolstra b5fcb5b67d Use the qemu-kvm wrapper 2013-07-31 14:53:35 +02:00
aszlig 69dccda3c0
VM builds: Update debian Wheezy image to v7.1.
Wheezy has been released on June 15th and on all mirrors the SHA256 hash
of Packages.bz2 has changed to reflect the new release, so let's update.

Here is the release announcement from Debian:

http://www.debian.org/News/2013/20130615

It also seems that the versioning scheme has changed in version 7.x, so
they seem to have switched to a two digit versioning scheme. This means,
that the attribute name "debian70..." should really be something like
"debian7...", but I'm keeping the attribute as-is to not break
references.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2013-07-30 05:59:59 +02:00
aszlig cd4b5e1a35
fillDiskWithDebs: Create fake start-stop-daemon.
This is needed in order to prevent services from starting while
populating the image with the contents of the .deb files. The procedure
used here is exactly the same as used in debootstrap.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2013-07-30 05:59:40 +02:00
Eelco Dolstra 59aca26975 * Use 'or'. 2013-07-15 14:33:27 +02:00
Rob Vermaas da7db90068 Add CentOS 6.4 to vm images 2013-07-05 17:52:42 +02:00
Rob Vermaas 51a22a8f8a Add loopback network device and localhost entry to /etc/hosts for VM tests. 2013-07-05 15:02:43 +02:00
Eelco Dolstra 9f5f30a7e6 Remove runInGenericVM
It's not used anywhere and probably doesn't work anymore.
2013-07-05 00:17:04 +02:00
Eelco Dolstra 0e386d0c13 VM builds: Use 9p/virtfs instead of CIFS
9p (with caching enabled) is much faster than CIFS and doesn't require
Samba or virtual networking.  For instance, building GNU Hello with
CIFS takes ~323s on my laptop, but with 9p it takes 54s.

More measurements will be needed to see if "cache=fscache" is really
faster than "cache=loose" (the former seems to be a little bit
faster).
2013-07-05 00:17:04 +02:00
Eelco Dolstra 9efe759dd8 VM builds: Increase the default memory size to 512 MB 2013-07-05 00:17:04 +02:00
Eelco Dolstra 2321f2d55c VM builds: Panic on OOM 2013-07-05 00:17:04 +02:00
Eelco Dolstra 5f8571405b VM builds: Use ext4 instead of ext2 2013-07-05 00:17:03 +02:00
Eelco Dolstra ed37a6b465 VM builds: Use qemu instead of the obsolete qemu-kvm 2013-07-05 00:17:03 +02:00
Eelco Dolstra 7afc1149d9 testRPMImage: Use x86_64 image 2013-07-04 18:27:26 +02:00
Eelco Dolstra 5cfa68ab50 Reinstate sec=none
Seems to be required, despite the subsequent sec=ntlm. But the NixOS
VM tests work fine without this flag :-S

http://hydra.nixos.org/build/5451901
2013-07-03 14:38:56 +02:00
Eelco Dolstra 75e34854f6 Remove redundant cifs option 2013-07-02 17:03:29 +02:00
Shea Levy dd42dd480b runInLinuxImage: Fix derivation overriding.
This only ever worked because runInLinuxVM happened to call
overrideDerivation, which itself erroneously passed arbitrarily-added
attributes to the new call to derivation.

Hopefully this time Eelco won't have to revert my change ;)

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-05-15 01:15:16 -04:00
Eelco Dolstra 9f9620f566 Doh 2013-05-13 22:04:33 +02:00
Eelco Dolstra 278b255388 Restore unintentionally deleted line 2013-05-13 18:15:02 +02:00
Eelco Dolstra 9df603b300 Use Linux 3.9 for VM builds
This seems to fix CIFS hangs like: http://hydra.nixos.org/build/4916655
2013-05-13 18:12:12 +02:00
Eelco Dolstra 6fb4c62f25 Fix some old Fedora builds
http://hydra.nixos.org/build/4949447
2013-05-13 17:47:20 +02:00
Eelco Dolstra 5815d18460 Add Fedora 18 2013-05-08 14:08:59 +02:00
Eelco Dolstra e7480d9cb7 Add Debian 7.0 2013-05-07 11:19:46 +02:00
Eelco Dolstra 1073187f78 Add Ubuntu 13.04 2013-05-03 14:01:32 +02:00
Eelco Dolstra fb600a5d99 Debian VM image: Update to 6.0.7 2013-03-15 12:48:35 +01:00
Eelco Dolstra b643316a42 VM builds: Use BusyBox
http://hydra.nixos.org/build/3510928
2012-12-18 16:57:11 +01:00
Lluís Batlle i Rossell 62fa1b8782 Adding ext4 to vmTools. Otherwise, nixos build-vm fails with 'useBootLoader'. 2012-12-16 21:43:35 +01:00
Eelco Dolstra 7f115621ca Add Ubuntu 12.10 2012-12-04 20:06:07 +01:00
Eelco Dolstra 85650db656 Update Debian Squeeze to 6.0.6 2012-12-04 19:58:21 +01:00
Eelco Dolstra b29d424115 Get deb-closure.pl to work with the latest dpkg 2012-12-04 19:56:31 +01:00
Eelco Dolstra 2ab46949cf Add universe repo for older Ubuntu releases 2012-09-13 17:15:58 -04:00
Eelco Dolstra fc35bed470 Fix incorrect version of Ubuntu Maverick (10.10) 2012-09-13 17:15:58 -04:00
Eelco Dolstra 20de8c8086 Make the "universe" repository available to the Ubuntu image generator
Also fix Ubuntu 12.04 name from "oneiric" to "precise".
2012-08-27 13:53:07 -04:00
Eelco Dolstra bacc6ab790 Remove trailing whitespace 2012-08-27 13:53:07 -04:00
Eelco Dolstra 395718e927 Create /dev/random and /dev/urandom in VMs
WWW::Curl needs this to prevent "Fatal: no entropy gathering module
detected".
2012-08-27 13:53:07 -04:00
Eelco Dolstra a54734e087 vmTools: Provide hook to add more Samba shares 2012-08-16 10:55:13 -04:00
Rob Vermaas 0cbfd8ff04 * update hashes for debian queeze
svn path=/nixpkgs/trunk/; revision=34316
2012-06-01 17:53:53 +00:00
Eelco Dolstra b6917d3b73 * Added Ubuntu 12.04.
svn path=/nixpkgs/trunk/; revision=34055
2012-05-11 02:04:14 +00:00
Eelco Dolstra 9aa30ba705 * Pass -cpu kvm64 to keep GMP from barfing.
svn path=/nixpkgs/trunk/; revision=33848
2012-04-19 18:44:02 +00:00
Eelco Dolstra 8b0bc7a745 * Add modules required by Linux 3.2.
svn path=/nixpkgs/trunk/; revision=33742
2012-04-11 07:45:12 +00:00
Ludovic Courtès 9d125d64bf VM: Upgrade to Debian 5.0.10.
svn path=/nixpkgs/trunk/; revision=33008
2012-03-12 08:13:10 +00:00
Ludovic Courtès 0a433261c5 VM: Make sure `smbd' is in QEMU's $PATH.
svn path=/nixpkgs/trunk/; revision=32987
2012-03-11 14:23:32 +00:00
Ludovic Courtès 5d159f8a06 runInGenericVM: Use whatever the current system is.
svn path=/nixpkgs/trunk/; revision=32970
2012-03-10 11:44:47 +00:00
Ludovic Courtès 2f01e58e61 vmTools: Use stdenv's glibc in `initrd-utils'.
svn path=/nixpkgs/trunk/; revision=32878
2012-03-08 09:40:45 +00:00
Eelco Dolstra ed325cd1ae * We accidentally lost xz support in rpm. This broke RPM builds
(http://hydra.nixos.org/build/2230623).

svn path=/nixpkgs/trunk/; revision=32847
2012-03-07 09:47:22 +00:00