Commit graph

74 commits

Author SHA1 Message Date
Sarah Brofeldt b256df4937 dockerTools: Use nix instead of nixUnstable 2018-10-01 09:51:52 +02:00
Graham Christensen fb2d153dac
dockerTools: test buildLayeredImage 2018-09-27 14:19:43 -04:00
Graham Christensen 4fe9006190 dockerTools.buildLayeredImage: init
Create a many-layered Docker Image.

Implements much less than buildImage:

 - Doesn't support specific uids/gids
 - Doesn't support runninng commands after building
 - Doesn't require qemu
 - Doesn't create mutable copies of the files in the path
 - Doesn't support parent images

If you want those feature, I recommend using buildLayeredImage as an
input to buildImage.

Notably, it does support:

 - Caching low level, common paths based on a graph traversial
   algorithm, see referencesByPopularity in
   0a80233487993256e811f566b1c80a40394c03d6
 - Configurable number of layers. If you're not using AUFS or not
   extending the image, you can specify a larger number of layers at
   build time:

       pkgs.dockerTools.buildLayeredImage {
         name = "hello";
         maxLayers = 128;
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

 - Parallelized creation of the layers, improving build speed.
 - The contents of the image includes the closure of the configuration,
   so you don't have to specify paths in contents and config.

   With buildImage, paths referred to by the config were not included
   automatically in the image. Thus, if you wanted to call Git, you
   had to specify it twice:

       pkgs.dockerTools.buildImage {
         name = "hello";
         contents = [ pkgs.gitFull ];
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

   buildLayeredImage on the other hand includes the runtime closure of
   the config when calculating the contents of the image:

       pkgs.dockerTools.buildImage {
         name = "hello";
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

Minor Problems

 - If any of the store paths change, every layer will be rebuilt in
   the nix-build. However, beacuse the layers are bit-for-bit
   reproducable, when these images are loaded in to Docker they will
   match existing layers and not be imported or uploaded twice.

Common Questions

 - Aren't Docker layers ordered?

   No. People who have used a Dockerfile before assume Docker's
   Layers are inherently ordered. However, this is not true -- Docker
   layers are content-addressable and are not explicitly layered until
   they are composed in to an Image.

 - What happens if I have more than maxLayers of store paths?

   The first (maxLayers-2) most "popular" paths will have their own
   individual layers, then layer #(maxLayers-1) will contain all the
   remaining "unpopular" paths, and finally layer #(maxLayers) will
   contain the Image configuration.
2018-09-26 17:54:14 -04:00
Graham Christensen 2bf0ee3b2b dockertools: tarsum: turn in to a buildInput 2018-09-26 15:50:04 -04:00
Graham Christensen aedc651903
dockerTools.buildImage: test that created=now makes an unstable date 2018-09-20 13:06:14 -04:00
Graham Christensen a32d7e0c74 dockerTools.buildImage: support impure dates
Because dates are an impurity, by default buildImage will use a static
date of one second past the UNIX Epoch. This can be a bit frustrating
when listing docker images in the CLI:

    $ docker image list
    REPOSITORY   TAG      IMAGE ID       CREATED        SIZE
    hello        latest   08c791c7846e   48 years ago   25.2MB

If you want to trade the purity for a better user experience, you can
set created to now.

    pkgs.dockerTools.buildImage {
      name = "hello";
      tag = "latest";
      created = "now";
      contents = pkgs.hello;

      config.Cmd = [ "/bin/hello" ];
    }

and now the Docker CLI will display a reasonable date and sort the
images as expected:

    $ docker image list
    REPOSITORY   TAG      IMAGE ID       CREATED              SIZE
    hello        latest   de2bf4786de6   About a minute ago   25.2MB
2018-09-20 18:26:02 +02:00
Jack Kelly af5eab6ea6 dockerTools.pullImage: correct default arch 2018-09-19 16:13:15 +10:00
Nick Novitski c58b11d229 dockerTools.pullImage: control OS and architecture 2018-07-27 12:29:31 -07:00
volth 52f53c69ce pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
Antoine Eiche d44b778d10 dockerTools.examples: explicitly set image tag to fix docker-tools tests
docker-tools tests load images without specifying any tag
value. Docker then uses the image with tag "latest" which doesn't
exist anymore since commit 39e678e24e.
2018-07-06 16:38:42 +02:00
Mathias Schreck 39e678e24e dockerTools.buildImage: add option to use nix output hash as tag 2018-07-06 15:15:09 +02:00
lewo 0644b4d948 dockerTools.pullImage: expose image* attributes (#41366)
Attributes `imageName` and `imageTag` are exposed if the image is
built by our Nix tools but not if the image is pulled. So, we expose
these attributes for convenience and homogeneity.
2018-06-03 22:58:23 +02:00
lewo 2e98e0c003
Merge pull request #40947 from samueldr/fix/34779
dockerTools: fixes extraCommands for mkRootLayer.
2018-05-24 21:22:31 +02:00
Samuel Dionne-Riel 902b0593be tests/docker-tools: Adds regression test for #34779 2018-05-24 12:23:51 -04:00
Antoine Eiche 8f71ce7e80 skopeo: 0.1.29 -> 0.1.30
Skopeo used by our docker tools was patched to work in the build
sandbox (it used /var/tmp which is not available in the sandbox).
Since this temporary directory can now be set at build time, we remove
the patch from our docker tools.
2018-05-24 15:33:52 +02:00
Samuel Dionne-Riel 60737bd319 dockerTools: fixes extraCommands for mkRootLayer.
The extraCommands was, previously, simply put in the body of the script
using nix expansion `${extraCommands}` (which looks exactly like bash
expansion!).

This causes issues like in #34779 where scripts will eventually create
invalid bash.

The solution is to use a script like `run-as-root`.

 * * *

Fixes #34779
2018-05-24 06:51:26 -04:00
aszlig 42a0b11450
dockerTools.pullImage: Fix build with sandboxing
Regression introduced in 736848723e.

This commit most certainly hasn't been tested with sandboxing enabled
and breaks not only pullImage but also the docker-tools NixOS VM test
because it doesn't find it's certificate path and also relies on
/var/tmp being there.

Fixing the certificate path is the easiest one because it can be done
via environment variable.

I've used overrideAttrs for changing the hardcoded path to /tmp (which
is available in sandboxed builds and even hardcoded in Nix), so that
whenever someone uses Skopeo from all-packages.nix the path is still
/var/tmp.

The reason why this is hardcoded to /var/tmp can be seen in a comment in
vendor/github.com/containers/image/storage/storage_image.go:

  Do not use the system default of os.TempDir(), usually /tmp, because
  with systemd it could be a tmpfs.

With sandboxed builds this isn't the case, however for using Nix without
NixOS this could turn into a problem if this indeed is the case.

So in the long term this needs to have a proper solution.

In addition to that, I cleaned up the expression a bit.

Tested by building dockerTools.examples.nixFromDockerHub and the
docker-tools NixOS VM test.

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @nlewo, @Mic92, @Profpatsch, @globin, @LnL7
2018-05-06 04:57:24 +02:00
Antoine Eiche 736848723e dockerTools.pullImage: Skopeo pulls images by digest
Skopeo is used to pull images from a Docker registry (instead of a
Docker deamon in a VM).

An image reference is specified with its name and its digest which is
an immutable image identifier (unlike image name and tag).

Skopeo can be used to get the digest of an image, for instance:
$ skopeo inspect docker://docker.io/nixos/nix:1.11 | jq -r '.Digest'
2018-05-02 21:32:20 +02:00
Jean-Philippe Braun 9751771c73 dockerTools.buildImage: add /nix/store with correct permissions
Fixes #38835.
2018-04-16 10:19:01 +02:00
Ryan Trinkle 1034aa8e9c
Merge pull request #25148 from obsidiansystems/docker-dirlinks
dockerTools: optionally preserve directory symlinks
2018-04-09 17:44:09 -04:00
Sarah Brofeldt 4874ce1701 dockerTools.tarsum: Fix upstream import 2018-03-26 18:47:31 +02:00
Antoine Eiche ac0c491836 dockerTools: add --sort=name options on all tar calls
This is to go to a reproducible image build.
Note without this options image are identical from the Docker point of
view but generated docker archives could have different hashes.
2018-03-13 13:46:47 +01:00
Antoine Eiche 346996ceec dockerTools: dereference hard links in tar archives
This is to improve image creation reproducibility. Since the nar
format doesn't support hard link, the tar stream of a layer can be
different if a dependency of a layer has been built locally or if it
has been fetched from a binary cache.

If the dependency has been build locally, it can contain hard links
which are encoded in the tar stream. If the dependency has been
fetched from a binary cache, the tar stream doesn't contain any hard
link. So even if the content is the same, tar streams are different.
2018-03-13 13:46:41 +01:00
Antoine Eiche e8f452f110 dockerTools: add an onTopOfPulledImage example
This allows to test if a pulled image can be updated by using our
Docker tools.
2018-03-13 11:59:22 +01:00
Antoine Eiche ce838e52b9 dockerTools.buildImage: do not add /nix/store in the tar stream
Since the /nix/store directory is not immutable, tar can fails if it
has to push it into the layer archive.

Fixes #34137.
2018-02-14 06:40:41 +01:00
John Ericson 888404f11b treewide: Fix deps in a few other fixed output derivations 2018-01-10 11:18:44 -05:00
Wei-Ming Yang 70e9b60b33
dockerTools.examples: correct a typo in comments
This commit is for correcting a typo in comments.
2018-01-01 16:13:40 +08:00
Ryan Trinkle ded1281f45
Merge branch 'master' into docker-dirlinks 2017-11-03 10:53:00 -04:00
Robin Gloster 20677fca59
dockerTools: fix hash to accomodate the pullImage revert 2017-09-28 14:09:49 +02:00
Robin Gloster 5c6dc717a6
Revert "dockerTools.pullImage: use skopeo to pull the image"
This reverts commit 01174c5f4d.

See https://github.com/NixOS/nixpkgs/pull/29302#issuecomment-332809092
for more information. This broke image format compatibility and
therefore amongst others mesos.
2017-09-28 14:09:49 +02:00
Robin Gloster dabb296c76
Revert "dockerTools.buildImage: Switch to the format image generated by Skopeo"
This reverts commit 35f205a4b6.

This does not use a standard format and by that breaks mesos
2017-09-28 13:01:34 +02:00
Antoine Eiche ff4d7f0fd2 dockerTools.examples.nix: set NIX_PAGER=cat environment variable 2017-09-25 09:39:15 +02:00
Antoine Eiche 35f205a4b6 dockerTools.buildImage: Switch to the format image generated by Skopeo
We were using 'Combined Image JSON + Filesystem Changeset Format' [1] to
unpack and pack image and this patch switches to the format used by the registry.

We used the 'repository' file which is not generated by Skopeo when it
pulls an image. Moreover, all information of this file are also in the
manifest.json file.
We then use the manifest.json file instead of 'repository' file. Note
also the manifest.json file is required to push an image with Skopeo.

Fix #29636

[1] 749d90e10f/image/spec/v1.1.md (combined-image-json--filesystem-changeset-format)
2017-09-23 13:17:07 +02:00
Antoine Eiche cb6fc52f99 dockerTools.buildImageWithNixDb: Make output paths valid and add gcroots
The database dump doesn't contain sha and size. This leads to invalid
path in the container. We have to fix the database by using
nix-store.
Note a better way to do this is available in Nix 1.12 (since the
database dump contains all required information).

We also add content output paths in the gcroots since they ca be used
by the container.
2017-09-20 20:14:29 +02:00
Antoine Eiche df589a438e dockerTools.buildImageWithNixDb: populate the Nix Db of the image Nix store
Currently, the contents closure is copied to the layer but there is no
nix database initialization. If pkgs.nix is added in the contents,
nix-store doesn't work because there is no nix database.

From the contents of the layer, this commit generates and loads the
database in the nix store of the container. This only works if there
is no parent layer that already have a nix store (to support several
nix layers, we would have to merge nix databases of parent layers).

We also add an example to play with the nix store inside the
container. Note it seems `more` is a missing dependency of the nix
package!
2017-09-20 20:14:24 +02:00
Antoine Eiche 01174c5f4d dockerTools.pullImage: use skopeo to pull the image
Before this patch, a VM was used to spawn docker that pulled the
VM. Now, the tool Skopeo does this job well so we can simplify our
dockerTools since we doesn't need Docker anymore:)

This also fixe the regression described in
https://github.com/NixOS/nixpkgs/issues/29271 : cntlm proxy doesn't
work in 17.09 while it worked in 17.03.

Note Skopeo doesn't produce the same output than docker pull so, we
have to update sha.
2017-09-17 08:26:02 +01:00
Antoine Eiche 132e790735 dockerTools.pullImage: change the docker deamon readiness mechanism
To wait for the docker deamon, curl requests are sent. However, if a
http proxy is set, it will respond instead of the docker daemon.
To avoid this, we send docker ps command instead of curl command.
2017-09-04 10:52:16 +02:00
Mathias Schreck 86d9b09c9b dockerTools: fix image json and manifest
The image json is not exactly the same as the layer json, therefore I
changed the implementation to use the `baseJson` which doesn’t include
layer specific details like `id`, `size` or the checksum of the layer.

Also the `history` entry was missing in the image json. I’m not totally
sure if this field is required, but a I got an error from a docker
registry when I’ve tried to receive the distribution manifest of an
image without those `history` entry:

GET: `http://<registry-host>/v2/<imageName>/manifests/<imageTag>`

```json
{
  "errors": [
    {
      "code": "MANIFEST_INVALID",
      "message": "manifest invalid",
      "detail": {}
    }
  ]
}
```

I’ve also used a while loop to iterate over all layers which should make
sure that the order of the layers is correct. Previously `find` was
used and I’m not sure if the order was always correct.
2017-08-03 11:52:03 +02:00
Antoine Eiche 9082b66096 dockerTools: fix permissions on base image
If the base image has been built with nixpkgs.dockerTools, the image
configuration and manifest are readonly so we first need to change
their permissions before removing them.

Fix #27632.
2017-07-31 13:52:35 +02:00
Antoine Eiche 9ee7e8b67e docker: generate the image configuration and manifest
This is required to push images to the Docker registry v2.
2017-07-26 23:05:29 +02:00
Antoine Eiche 8a431e13b5 docker: Remove ./ pattern when packing an image
Elements in images tar.gz generated by docker don't start by './'.
2017-07-26 23:05:29 +02:00
Antoine Eiche 0a4c43065c docker: do not import configuration and manifest from the base image
Fix #27632.
2017-07-26 20:01:41 +02:00
Antoine Eiche e28d817c9a docker: lowercase image name and tag
The docker loading (docker 1.12.6) of an image with uppercase in the
name fails with the following message:
invalid reference format: repository name must be lowercase
2017-07-25 10:47:51 +02:00
Daiderd Jordan 90ff6b1d03 Merge pull request #27017 from LnL7/docker-pure-layer
docker-tools: set user/group when creating a pure layer
2017-07-15 12:22:53 +02:00
Sarah Brofeldt 9e61958399 tarsum: Fix source path in docker.src 2017-07-13 23:09:04 +02:00
Daiderd Jordan 83fbc0f035
docker-tools: fixup permssions for extraCommands 2017-07-08 13:57:04 +02:00
Daiderd Jordan dde5865140
docker-tools: set group/owner when creating a pure layer 2017-07-01 17:08:48 +02:00
Michael Fellinger 54419f3016 using inherited instead 2017-06-27 22:22:08 +02:00
Michael Fellinger 704e04b108 dockerTools.buildImage: configurable timestamp
This way not all images have to be from 47 years ago, making it much easier to find the one you're looking for.
2017-06-27 22:11:17 +02:00
Matej Cotman 8d3cb0af9e docker tools: fix pull image function 2017-05-26 18:48:16 +02:00