Commit graph

141 commits

Author SHA1 Message Date
lewo a498da343a
Merge pull request #87154 from utdemir/buildimage-optimizations
Some performance optimizations to dockerTools.build{,Layered}Image
2020-05-19 15:39:25 +02:00
zowoq 42232493a3 dockerTools: pass insecure-policy and tmpdir to skopeo 2020-05-16 08:46:13 +10:00
lewo df928fafd4
Merge pull request #87725 from nlewo/fix-dockertools
dockerTools.examples: fix the nginx image
2020-05-15 16:54:09 +02:00
Antoine Eiche 9ac444d80e dockerTools.examples: fix the nginx image
Nginx now requires directories `/var/log/nginx` and `/var/cache/nginx`
to be existing.

The objective is to fix the test `nixosTests.docker-tools`.
2020-05-13 10:29:53 +02:00
Utku Demir f5a90a7aab
dockerTools.buildImage: Preserve environment variables from the parent image 2020-05-08 21:49:16 +12:00
Utku Demir f12346d493
dockerTools: Calculate tarsum's on the fly
Calculating the tarsum after creating a layer is inefficient, since
we have to read the tarball we've just written from the disk.

This commit simultaneously calculates the tarsum while creating the
tarball.
2020-05-07 11:50:07 +12:00
Utku Demir 69f6294724
dockerTools.buildLayeredImage: Avoid appending to tarballs when building layers
Appending to an existing tar archive repeatedly seems to be a quadratic
operation, since tar seems to traverse the existing archive even using
the `-r, --append` flag. This commit avoids that by passing the list of
files to a single tar invocation.
2020-05-07 11:50:00 +12:00
adisbladis fafb127947
dockertools: Add a buildLayeredImageWithNixDb function
This is analogous to buildImageWithNixDb but instead uses
buildLayeredImage under the hood.
2020-05-02 15:59:39 +01:00
Simonas Kazlauskas dde3e8edf4 nix-prefetch-docker: Add jq into path
Otheriwse running the tool fails with

    line 41: jq: command not found
2020-04-27 22:21:55 +03:00
Robert Hensing 6dab1b50a6 buildLayeredImage: Allow empty store, no paths to add
This is useful when buildLayeredImage is called in a generic way
that should allow simple (base) images to be built, which may not
reference any store paths.
2020-02-28 14:59:04 +01:00
Antoine Eiche baa78de594 nixosTests.docker-tools: add bulk-layer test
A regression test for https://github.com/NixOS/nixpkgs/issues/78744.
2020-02-14 09:26:26 +01:00
Richard Wallace 3b65b3f6d6 dockerTools.buildLayeredImage: store all paths passed in final layer
Fixes #78744

My previous change broke when there are more packages than the maximum
number of layers. I had assumed that the `store-path-to-layer.sh` was
only ever passed a single store path, but that is not the case if
there are multiple packages going into the final layer. To fix this, we
loop through the paths going into the final layer, appending them to the
tar file and making sure they end up at the right path.
2020-02-14 09:26:26 +01:00
Silvan Mosberger 0a351c3f65
dockerTools.*: Assertion against building for Darwin (#77952)
dockerTools.*: Assertion against building for Darwin
2020-01-31 21:17:40 +01:00
Antoine Eiche 01a68479cc dockerTools.buildLayeredImage: assert maxLayers > 1
Since a layer is reserved for "customization", the image can not
contains less than 2 layers.

The user gets the following message at evaluation:

    nix-instantiate nixos/tests/docker-tools.nix
    trace: the maxLayers argument of dockerTools.buildLayeredImage function must be greather than 1 (current value: 1)
2020-01-30 21:09:52 +01:00
Antoine Eiche 283bcc1003 dockerTools.buildLayeredImage: fix image with only 2 layers
A test is also added to ensure an image with 2 layers can be built.
2020-01-30 21:09:44 +01:00
Silvan Mosberger b6a9211bf4
dockerTools.*: Assertion against building for darwin
Building a docker image with darwin binaries just yields a confusing
error when ran:

  standard_init_linux.go:211: exec user process caused "exec format error"

This change prevents people from building such images in the first place
2020-01-20 19:14:12 +01:00
Antoine Eiche da261e3631 dockerTools.buildLayeredImage: fix typo in comments 2020-01-11 09:02:30 +01:00
Richard Wallace 3be767593b dockerTools.buildLayeredImage: fix building layered images in parallel
when tar'ing store paths into layered archives when building layered
images, don't use the absolute nix store path so that tar won't complain
if something new is added to the nix store

when building the final docker image, ignore any file changes tar
detects in the layers. they are all immutable and the only thing that
might change is the number of hard links due to store optimization
2019-12-30 14:47:11 -07:00
Graham Christensen 64453c8dbd
Merge pull request #75781 from grahamc/dockertools/remove-implementation-detail-layers
dockertools.buildLayeredImage: remove implementation detail layers
2019-12-17 16:03:11 -05:00
Graham Christensen 75b8f3820d
Merge pull request #75779 from grahamc/dockertools/fixup-max-layers
dockerTools.buildLayeredImage: update maxlayers from 24 to 100 to match documentation
2019-12-16 14:11:36 -05:00
Graham Christensen 9c02760855
dockerTools.buildLayeredImage: update maxlayers from 24 to 100 to match documentation
mkManyPureLayers already was changed, and this function was not updated.
2019-12-16 13:14:21 -05:00
Graham Christensen 77452740c6
docker examples: Drop unneeded contents list 2019-12-16 13:11:27 -05:00
Graham Christensen 12e2416380
dockerTools.buildLayeredImage: Exclude top level implementation detail layers 2019-12-16 13:03:15 -05:00
Graham Christensen 700f4c5388
dockerTools.buildLayeredImage: prepare to exclude some paths
Without changing behavior, since this code is fiddly, make it possible
to add a filtering step before packaging individual paths.
2019-12-16 12:57:04 -05:00
Graham Christensen aec80dddc0
dockerTools.buildLayeredImage: pass a list of closures to mkManyPureLayers so it can exclude the top-most level
Before, every docker image had three extra layers:

1. A `closure` layer which is an internal implementation detail of
   calculating the closure of the container
2. a `name-config.json` layer which is the images' run-time
   configuration, and has no business being *in* the image as a layer.
3. a "bulk-layers" layer which is again and implementation detail
   around collecting the image's closure.

None of these layers need to be in the final product.
2019-12-16 12:48:05 -05:00
Graham Christensen f6d75f550e
dockerTools.buildLayeredImage: tweak formatting on contentsEnv 2019-12-16 12:36:45 -05:00
tomberek 81b0a20dfa buildImageWithNixDb: export USER (#74959)
dockerTools.buildImageWithNixDb: export USER

Changes to Nix user detection (./src/nix-channel/nix-channel.cc#L-166)
cause this function to error. Exporting USER fixes this.
2019-12-07 10:06:42 +01:00
Antoine Eiche 09a669478f dockerTools.examples.nix: set USER in the container config
A USER is required by Nix.
See 9348f9291e/src/libutil/util.cc (L478).
2019-12-05 09:45:51 +01:00
Nick Spinale 8166bc934b build-support/docker: set default image arch to host arch
The architecture of an image should default to the architecture for
which that image is being composed or pulled. buildPackages.go.GOARCH is
an easy way to compute that architecture with the correct terminology.
2019-09-16 14:22:30 +00:00
Graham Christensen 8adaae2df2
dockertools.buildLayeredImage: default layer count up to 100 2019-08-12 19:59:40 -04:00
Sarah Brofeldt 19abdb765b
Merge pull request #63865 from dingxiangfei2009/dockertools-cross-compilation
dockerTools: use skopeo on the right platform
2019-07-31 20:11:18 +02:00
Ding Xiang Fei ceab72e21b dockerTools: use skopeo on the right platform 2019-07-02 14:53:24 +08:00
worldofpeace 3f4a353737 treewide: use dontUnpack 2019-07-01 04:23:51 -04:00
volth f3282c8d1e treewide: remove unused variables (#63177)
* treewide: remove unused variables

* making ofborg happy
2019-06-16 19:59:05 +00:00
Matthew Bauer 4e5c9b8cf4
Merge pull request #54921 from grabango/master
Use nativeBuildInputs for building Docker images
2019-06-10 21:38:47 -04:00
Daniël de Kok 3b0d5b5ed3 nixos/tests/docker-tools: check layer sharing with buildLayeredImage
Adapted from grahamc's blog post on layered Docker images in Nix:

https://grahamc.com/blog/nix-and-layered-docker-images
2019-06-06 15:03:08 +02:00
Daniël de Kok d7f3186b48 dockerTools.buildLayeredImage: restore layer sharing
PR #58431 added /nix/store to each layer.tar. However, the timestamp was
not explicitly set while adding /nix and /nix/store to the archive. This
resulted in different SHA256 hashes of layer.tar between image builds.

This change sets time and owner when tar'ing /nix/store.
2019-06-06 07:40:20 +02:00
Antoine Eiche a5a5820048 dockerTools: Fix the layer order
The layer order was not correct when a parent image was used: parent
image layers were above the new created layer.

This commits simplifies the code related to layer ordering. In
particular, layers in `layer-list` are ordered from bottom-most to
top-most. This is also the order of layers in the `rootfs.diff_ids`
attribute of the image configuration.
2019-05-07 16:52:25 +02:00
Antoine Eiche 5ef1223f30 nixos/tests/docker-tools: verify order of layers in stacked images 2019-05-07 16:52:13 +02:00
rnhmjoj 5ea4160b33
treewide: fixup bcf54ce
fix the executable bit for scripts installed with substituteAll
and some remaining shebangs.
2019-04-12 09:09:22 +02:00
lewo a8beff987d
Merge pull request #58350 from xtruder/pkgs/dockerTools/storePathToLayer/runtimeShell
dockerTools: storePathToLayer use runtimeShell in script
2019-04-06 19:01:19 +02:00
Jaka Hudoklin c087b608e8
Merge pull request #58360 from xtruder/pkgs/dockerTools/nix-prefetch-docker
dockerTools: add nix-prefetch-docker script
2019-04-06 14:44:19 +02:00
Jaka Hudoklin c1f0d19785
dockerTools: add nix-prefetch-docker script 2019-04-06 14:16:40 +02:00
lewo dc3ed336df
Merge pull request #58345 from xtruder/pkgs/dockerTools/pullImage/finalImageName
dockerTools: add finalImageName parameter for pullImage
2019-03-28 16:25:01 +01:00
Alex Biehl 1b1e23024b Strip leading ./ in customization layer 2019-03-27 12:08:50 +01:00
Alex Biehl 096e3cbdec Add /nix/store to each layer.tar
To be totally consistent with the way Docker builds images we need to
include the /nix/store in the layer tarballs first.
2019-03-27 11:35:27 +01:00
Jaka Hudoklin 468df177c4
dockerTools: add finalImageName parameter for pullImage 2019-03-26 19:35:14 +01:00
Jaka Hudoklin 5d429f6822
dockerTools: storePathToLayer use runtimeShell in script 2019-03-26 11:07:24 +01:00
Antoine Eiche fe6860800b dockerTools.buildImage.runAsRoot: preserve layers ordering at image unpacking
This patch preserves the ordering of layers of a parent image when the
image is unpacked.

Fixes #55290
2019-03-12 12:04:22 +01:00
Antoine Eiche af23d1e2e7 nixos/test/docker-tools: fix Nix image digest 2019-03-06 09:26:33 +01:00