Commit graph

55 commits

Author SHA1 Message Date
Casey Ransom c3f28aed94 make-disk-image: Reduce forking
The disk image calculator was using find + exec forking du for every
file in the disk image, making it very slow. Change du to accept files,
nul delimeted on stdin to speed it back up.

Before change:
nix-build nixos/tests/image-contents.nix  9.71s user 1.06s system 8% cpu 2:13.11 total

After change:
nix-build nixos/tests/image-contents.nix  9.93s user 1.23s system 21% cpu 51.601 total
2021-06-21 22:58:07 -04:00
Lassulus ee04d772e4
Merge pull request #120489 from samueldr/fix/make-disk-image-auto-size
Fix make disk image automatic size
2021-04-26 10:34:15 +02:00
Martin Schwaighofer f20ae954d5 make-disk-image: fix broken EFI image builds
Work around missing /dev files inside runInLinuxVM by creating a
symlink before calling nixos-enter.

This fixes https://github.com/NixOS/nixpkgs/issues/93381.
I ran into this issue when trying to create a VMware image that boots from EFI.

Thanks @colemickens for reporting this and @danielfullmer for fixing the same thing in in qemu-vm.nix (37676e77cb) and explaining what the issue was.
2021-04-26 01:12:10 +02:00
Samuel Dionne-Riel 7b8b3fab6d make-disk-image: Round image size to the next mebibyte
This ensures the following gptfdisk warning won't happen:

```
Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
```

Additionally, helps towards aligning the partition to be more optimal
for the underlying storage.

It is actually impossible to align for the actual underlying storage
optimally because we don't know what the block device will be!

But aligning on 1MiB should help.
2021-04-25 15:24:45 -04:00
Samuel Dionne-Riel ba666011a6 make-disk-image: Account for reserved disk space
This is a bit of a thorny issue. See, the actual `diskSize` variable is
for the *total* disk size, not for the filesystem!

The automatic numbers are meant to compute the *filesystem* required
space. So we have to add any other reserved space!

We have different requirements for reserved space. E.g. there could be
none (when it's actually a filesystem image). There could also be 1MiB
for alignment for an MBR image, legacy+gpt needs 2MiB, then GPT with an
ESP ("bootSize") needs to take the boot partition and GPT size into
account too!

Though luckily(?) for this latter situation we can cheat! As noted in the
change, `bootSize` is NOT the boot partition size. It is actually the
offset where the target filesystem starts.
2021-04-24 14:49:05 -04:00
Samuel Dionne-Riel 9b18a78c73 make-disk-image: Account for the ext4 reserved space
Reserved space includes:

 - inodes space in use (2 blocks per)
 - about 5.2% of the space

The 5.2% reserved space was computed empirically when working on a
previous EXT4 image builder. It seems to stabilize around 5% even for
much larger filesystems.
2021-04-24 14:49:04 -04:00
Samuel Dionne-Riel 05c13a03e2 make-disk-image: Get proper size for automatic size
On some filesystems, `du` without `--apparent-size` will not give the
actual size for a file. Using `--apparent-size` will give us the actual
file size.

Though, this is not actually correct still. 1000 × 1 bytes is not 1000
bytes. It is 1000 × ceil(filesize/blockSize)*blockSize.

So instead of adding up the actual file sizes. We are adding up the
block sizes.

Note that this also changes the builder to work with *bytes*, rather
than with any other units. Doing maths on bytes is less likely to go
awry than doing it on other units.
2021-04-24 14:49:04 -04:00
Jörg Thalheim e5ce05cc1e
Merge pull request #67493 from clefru/make-disk-image 2021-01-29 07:18:50 +00:00
clefru 60c8cf4e02
Update nixos/lib/make-disk-image.nix
Co-authored-by: Alyssa Ross <hi@alyssa.is>
2021-01-22 19:58:23 +01:00
Francesco Gazzetta 6f21ae7a62 nixos/lib/make-disk-image.nix: support content mode and ownership 2020-12-20 21:22:47 +01:00
Graham Christensen bc49a0815a
utillinux: rename to util-linux 2020-11-24 12:42:06 -05:00
Graham Christensen d78aa080f5
make-disk-image: support legacy+gpt 2020-10-30 15:50:24 -04:00
Emery Hemingway 3f922834b8 nixos/lib/make-disk-image.nix: abritary format input
Pass unrecognized format types as the output file extension to
qemu-img. The motivation is support for "vdi" output.
2020-07-31 15:33:08 +02:00
lassulus 883a6079fd make-disk-image: add hybrid and dynamic sized images 2020-07-27 19:45:55 +02:00
lassulus 8d05772134
nixos make-disk-image: fix permissions of /build
This was broken in 460c0d6 (PR #90431); now the nixos-unstable channel
should get unblocked.
vcunat modified this commit to use env-var instead of hardcoding /build
2020-07-10 10:31:26 +02:00
Emery Hemingway b8a9e3eea1 nixos/lib/make-disk-image.nix: VDI support 2020-06-13 11:08:48 +05:30
Eelco Dolstra 39c0e489d3
Revert "make-disk-image: replace nixos-install by nix-env calls"
This reverts commit e9bf955fd6. We use
nixos-install to ensure that make-disk-image produces the same result
as a regular installation (9802da517f)
and to reduce code duplication. If there is something broken in
nixos-install, it should be fixed there.
2019-10-07 17:21:28 +02:00
Christian Kögler e9bf955fd6 make-disk-image: replace nixos-install by nix-env calls
nixos-install can only be called from a machine
2019-09-29 19:57:19 +02:00
Clemens Fruhwirth f664823a1a make-disk-image.nix: Handle cptofs error explicitly
Because the copy process inside the VM does not reliably
give "No space" error message leaving the user wondering what
went wrong:

unable to create directory /mnt/0000fe01///nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale/ro: Success
[    5.462365] reboot: Restarting system
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale/ro, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7, aborting
error processing entry /build/root/nix/store, aborting
error processing entry /build/root/nix, aborting
builder for '/nix/store/fsdvqxq92iai7f3w8wcsncgfwag7cj2l-libvirtd-ssh-image.drv' failed with exit code 228
2019-08-26 13:09:53 +02:00
Travis Athougies a66ef3aa3d Add options to build disk image function (#50239)
* add options to build disk image function

* Revert suffix changes
2019-01-24 01:47:55 +02:00
Matthew Bauer f05d8f31ec make-disk-image: use filterSource instead of cleanSource
cleanSource does not appear to work correctly in this case. The path
does not get coerced to a string, resulting in a dangling symlink
produced in channel.nix.  Not sure why, but this
seems to fix it.

Fixes #51025.

/cc @elvishjericco
2019-01-07 16:28:50 -06:00
volth 2e979e8ceb [bot] nixos/*: remove unused arguments in lambdas 2018-07-20 20:56:59 +00:00
Eelco Dolstra cc2eeef4ab
Fix installing the Nixpkgs channel on the installation media
And test that it got installed correctly.
2018-02-27 19:58:23 +01:00
Eelco Dolstra d12c9911df
Merge remote-tracking branch 'origin/master' into nix-2.0 2018-02-22 17:28:51 +01:00
Jan Malakhovski 2e6b796761 nixos: rename config.system.nixos* -> config.system.nixos.* 2018-02-18 12:56:30 +00:00
Eelco Dolstra 9802da517f
make-disk-image.nix: Use nixos-install again
Since nixos-install doesn't require any special privileges anymore,
this Just Works. No more need for fakeroot / nixos-prepare-root.
2018-02-07 19:39:13 +01:00
Tuomas Tynkkynen 962e79ef32 nixos/make-disk-image.nix: Support EFI images
- Add a new parameter `imageType` that can specify either "efi" or
  "legacy" (the default which should see no change in behaviour by
  this patch).

- EFI images get a GPT partition table (instead of msdos) with a
  mandatory ESP partition (so we add an assert that `partitioned`
  is true).

- Use the partx tool from util-linux to determine exact start + size
  of the root partition. This is required because GPT stores a secondary
  partition table at the end of the disk, so we can't just have
  mkfs.ext4 create the filesystem until the end of the disk.

- (Unrelated to any EFI changes) Since we're depending on the
  `-E offset=X` option to mkfs which is only supported by e2fsprogs,
  disallow any attempts of creating partitioned disk images where
  the root filesystem is not ext4.
2018-01-22 11:18:23 +02:00
Tuomas Tynkkynen 758b4c1ea4 make-disk-image.nix: Really fix write permissions in the store
I think the current one applies the -exec only to those that match
'-type d'. Let's switch it to something that humans can understand...
2017-12-20 17:28:46 +02:00
Tuomas Tynkkynen c9f71974f8 make-disk-image.nix: Remove write permissions from /nix/store
Fakeroot seems to always give the owner write bit to any files touched
inside it (presumably to easily simulate the fact that root can still
modify such files). So do an explicit chmod to remove them.

This should finally solve #32242 after the EC2 images are regenerated
with this change.

https://hydra.nixos.org/build/66143116
2017-12-17 01:19:05 +02:00
Tuomas Tynkkynen 0d27df280f build-support/vm: Use devtmpfs, not static device nodes
In 2017, there is no reason to create a static /dev.
2017-12-12 14:31:50 +02:00
Orivej Desh 470a7752d8 make-disk-image: fix eval 2017-11-13 14:56:30 +00:00
Orivej Desh 27f5cc959b make-disk-image: embed compression switch into the format name 2017-11-12 18:39:36 +00:00
Orivej Desh 608de48fe4 make-disk-image: add compressed option to compress qcow2 images 2017-11-12 18:28:06 +00:00
Orivej Desh a2d3ee77b8 make-disk-image: clean up 2017-11-12 18:24:26 +00:00
Orivej Desh a84ce72fdc make-disk-image: do not use faketime
Since https://github.com/lkl/linux/pull/394 cptofs preserves the source time,
which is 1970-01-01T00:00:01Z for /nix/store and recent for other files.

This reverts commit f5b3f2c5a7.
2017-11-11 20:57:14 +00:00
Orivej Desh f218ef647c make-disk-image: clean up 2017-11-11 20:24:26 +00:00
Dan Peebles f5b3f2c5a7 make-disk-iamge: use faketime to get a consistent timestamp
This fixes #28768 because during an image build, Nix sees bad store
timestamps and attempts to fix them, but can't fix them on a running
system (due to being inside a builder). Since timestamps on the store
are supposed to be 1 anyway, if we fix this, that fixes image building
inside booted images made this way.

Note that this adds quite a bit of noise to the output, because running
`cptofs` under `faketime` causes a bunch of seemingly spurious error
messages and my attempts to suppress them all failed. We'll fix it when
`cptofs` gets a native timestamp preservation feature.
2017-11-10 05:15:37 +00:00
Bjørn Forsman 0ff4bb5f87 nixos: run parted with --script option
-s, --script: never prompts for user intervention

Sometimes the NixOS installer tests fail when they invoke parted, e.g.
https://hydra.nixos.org/build/62513826/nixlog/1. But instead of exiting
right there, the tests hang until the Nix builder times out (and kills
the build). With this change the tests would instead fail immediately,
which is preferred.

While at it, use "parted --script" treewide, so nobody gets build
timeout due to parted error (or misuse). (Only nixos/ use it, and only
non-interactive.)

A few instances already use the short option "-s", convert them to long
option "--short".
2017-10-14 15:29:02 +02:00
Dan Peebles e2589b3ca2 Deduplicate some filterSource invocations
This version should have more conventional regexes that work across many
platforms and regex engines. This is an issue because up until Nix 1.11,
Nix called out to the libc regex matcher, which behaved differently on
Darwin and Linux. And in Nix 1.12, we're moving to std::regex which will
also behave differently here.

And yes, I do actually evaluate make-disk-image.nix on Darwin ;)
2017-08-29 20:27:04 -04:00
Dan Peebles 5f372ef67f amazon-image: support VPC/VHD as an output format for the builder 2017-08-10 22:57:26 +00:00
Dan Peebles f1708a9d7d make-disk-image: change to be less VM-centric
This changes much of the make-disk-image.nix logic (and thus most NixOS
image building) to use LKL to set up the target directory structure rather
than a Linux VM. The only work we still do in a VM is less IO-heavy stuff
that while still time-consuming, is less of the overall load. The goal is
to kill more of that stuff, but that will require deeper changes to NixOS
activation scripts and switch-to-configuration.pl, and I don't want to
bite off too much at once.
2017-04-24 02:30:00 +00:00
Dan Peebles e798f573f0 make-disk-image.nix: set last fsck time on ext4 images to enable resize-on-startup 2017-02-26 02:02:22 +00:00
Dan Peebles 49641e0de0 make-disk-image.nix: support additional filesystem contents
This makes make-disk-image.nix slightly more consistent with other image
builders we have. Unfortunately I duplicated some code in doing so, but
this is temporary duplication on the path to consolidating everything.
See https://github.com/NixOS/nixpkgs/issues/23052 for more details on that.

I'm also exposing the option in the amazon-image.nix maintainer module.
2017-02-22 23:49:49 +00:00
Domen Kožar b6363c7bc8 make-disk-image: make store validity fix optional
This is useful for EC2 AMI generation to speedup the process.

In my case it removes 13min out of 45min when generating an image
on EC2.
2016-12-07 13:30:20 +01:00
Domen Kožar e5cca82d79 make-disk-image: run tune2fs after umount to skip fsck
tune2fs marks the filesystem as clean to prevent resize2fs from
complaining.

But we were invoking it before we mounted the filesystem, so the
counters would increase to 1 and it broke the functionality.

By moving the call after the mount, I have confirmed it works by:

   $ nix-build nixos/tests/ec2.nix

cc @rbvermaa @edolstra
2016-12-07 13:30:20 +01:00
obadz 24f8cf08cc nixos/lib/make-disk-image: refactor to use nixos-install
- Replace hand-rolled version of nixos-install in make-disk-image by an
  actual call to nixos-install
- Required a few cleanups of nixos-install
- nixos-install invokes an activation script which the hand-rolled version
  in make-disk-image did not do. We remove /etc/machine-id as that's
  a host-specific, impure, output of the activation script

Testing:

nix-build '<nixpkgs/nixos/release.nix>' -A tests.installer.simple passes

Also tried generating an image with:

nix-build -E 'let
    pkgs = import <nixpkgs> {};
    lib = pkgs.lib;
    nixos = import <nixpkgs/nixos> {
      configuration = {
        fileSystems."/".device = "/dev/disk/by-label/nixos";
        boot.loader.grub.devices = [ "/dev/sda" ];
        boot.loader.grub.extraEntries = '"''"'
          menuentry "Ubuntu" {
             insmod ext2
             search --set=root --label ubuntu
             configfile /boot/grub/grub.cfg
          }
        '"''"';
      };
    };
  in import <nixpkgs/nixos/lib/make-disk-image.nix> {
    inherit pkgs lib;
    config = nixos.config;
    diskSize = 2000;
    partitioned = false;
    installBootLoader = false;
  }'

Then installed the image:
$ sudo df if=./result/nixos.img of=/dev/sdaX bs=1M
$ sudo resize2fs /dev/disk/by-label/nixos
$ sudo mount /dev/disk/by-label/nixos /mnt
$ sudo mount --rbind /proc /mnt/proc
$ sudo mount --rbind /dev /mnt/dev
$ sudo chroot /mnt /nix/var/nix/profiles/system/bin/switch-to-configuration boot

[ … optionally do something about passwords … ]

and successfully rebooted to that image.

Was doing all this from inside a Ubuntu VM with a single user nix install.
2016-08-16 15:31:16 +01:00
obadz 760b2b9048 nixos/make-disk-image: add ability to defer bootloader install until image has been flashed 2016-08-15 20:01:55 +01:00
Tuomas Tynkkynen 60f5659dad treewide: Use correct output in ${config.nix.package}/bin 2016-04-25 16:44:37 +02:00
Eelco Dolstra 0d3738cdcc Fix the EC2 test
We now generate a qcow2 image to prevent hitting Hydra's output size
limit. Also updated /root/user-data -> /etc/ec2-metadata/user-data.

http://hydra.nixos.org/build/33843133
2016-03-30 21:50:23 +02:00
Eelco Dolstra 5cc7bcda30 Combine OVA generation steps
Previously this was done in three derivations (one to build the raw
disk image, one to convert to OVA, one to add a hydra-build-products
file). Now it's done in one step to reduce the amount of copying
to/from S3. In particular, not uploading the raw disk image prevents
us from hitting hydra-queue-runner's size limit of 2 GiB.
2016-03-15 14:15:12 +01:00