Commit graph

50 commits

Author SHA1 Message Date
Robert Hensing d155b8c438 nixosTests.docker-tools: Fix nixFromDockerHub example sha
For https://github.com/NixOS/nixpkgs/pull/125211 I tried to test
the fetcher with

    nix-build -A dockerTools.examples.nixFromDockerHub --option substitute false

But it failed. I haven't figured out the cause, but the outputs
match, so it's probably the hashing method (flat/recursive) that
changed at some point. (The names did match.)
2021-06-01 15:13:03 +02:00
Robert Hensing 5259d66b74 dockerTools: Allow omitting all store paths
Adds includeStorePaths, allowing the omission of the store paths.
You generally want to leave it on, but tooling may disable this
to insert the store paths more efficiently via other means, such
as bind mounting the host store.
2021-05-26 15:11:42 +02:00
Viktor Kronvall 5caed960d3 dockerTools: add merged example images 2021-04-08 00:22:59 +09:00
Robert Hensing 58b21dea78
Merge pull request #116749 from vroad/docker-layered-image-fakeroot
dockerTools.streamLayeredImage: add fakeRootCommands option
2021-04-07 15:02:24 +02:00
vroad 63e7c4186f dockerTools.streamLayeredImage: Add test for fakeRootCommands 2021-04-07 18:11:02 +09:00
Louis Blin aae8588182 dockerTools.buildLayeredImage: support fromImage
It is now possible to pass a `fromImage` to `buildLayeredImage` and
`streamLayeredImage`, similar to what `buildImage` currently supports.

This will prepend the layers of the given base image to the resulting
image, while ensuring that at most `maxLayers` are used. It will also
ensure that environment variables from the base image are propagated
to the final image.
2021-03-23 14:50:42 +00:00
Louis Blin 419a4fa596 dockerTools.buildLayeredImage: image names with registry/ prefix
When using `buildLayeredImage`, it is not possible to specify an image
name of the form `<registry>/my/image`, although it is a valid name.

This is due to derivations under `buildLayeredImage` using that image
name as their derivation name, but slashes are not permitted in that
context.

A while ago, #13099 fixed that exact same problem in `buildImage` by
using `baseNameOf name` in derivation names instead of `name`. This
change does the same thing for `buildLayeredImage`.
2021-03-10 17:44:24 +00:00
Sarah Brofeldt ffe5ff6009 dockerTools: Test buildLayeredImage with symlinks
This exercises layer creation in face of store path symlinks, ensuring
they are not dereferenced, which can lead to broken layer tarballs
2021-01-04 21:44:47 +01:00
Florian Klink f7ee2706c2 dockerTools.fakeNss: init
This provides a /etc/passwd and /etc/group that contain root and nobody.

Useful when packaging binaries that insist on using nss to look up
username/groups (like nginx).

The current nginx example used the `runAsRoot` parameter to setup
/etc/group and /etc/passwd (which also doesn't exist in
buildLayeredImage), so we can now just use fakeNss there and use
buildLayeredImage.
2020-12-02 14:56:07 +01:00
adisbladis b7b22c5814
dockerTools: Always cross compile for another arch in the cross example
The example fails to build on aarch64, so lets cross build for gnu64.
2020-11-20 12:57:58 +01:00
adisbladis 5357abf49a
dockerTools: Add cross example 2020-11-19 18:13:21 +01:00
Utku Demir ae82f81bfa
dockerTools.streamLayeredImage: Store the customisation layer as a tarball
This fixes as issue described here[1], where permissions set by 'extraCommands'
were ignored by Nix.

[1] https://github.com/NixOS/nixpkgs/pull/91084#issuecomment-669834938
2020-09-04 16:53:23 +12:00
Johan Thomsen f5db415e2f nixos/tests/dockerTools: add test for running non-root containers with buildLayeredImage
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-07-31 10:14:07 +02:00
Robert Hensing 8c0459f611 dockerTools: Always set imageTag attribute
The image tag can be specified or generated from the output hash.
Previously, a generated tag could be recovered from the evaluated
image with some string operations.

However, with the introduction of streamLayeredImage, it's not
feasible to compute the generated tag yourself.

With this change, the imageTag attribute is set unconditionally,
for the buildImage, buildLayeredImage, streamLayeredImage functions.
2020-07-11 16:58:25 +02:00
Jos van Bakel 66737168a4
dockerTools.buildLayeredImage: fix created=now 2020-07-09 09:34:18 +02:00
Utku Demir 06db331922
dockerTools: Verify nix-store contents on buildLayeredImage test 2020-07-06 16:59:58 +12:00
Utku Demir cc46362929
dockerTools: Support files directly under /nix/store
Also makes sure that the files inside a layer added in a sorted order
to make the results more deterministic.
2020-07-04 22:00:57 +12:00
Robert Hensing caf47063b4 dockerTools: test that tar keeps nix binary symlinks intact 2020-06-19 10:08:34 +02:00
lewo df928fafd4
Merge pull request #87725 from nlewo/fix-dockertools
dockerTools.examples: fix the nginx image
2020-05-15 16:54:09 +02:00
Antoine Eiche 9ac444d80e dockerTools.examples: fix the nginx image
Nginx now requires directories `/var/log/nginx` and `/var/cache/nginx`
to be existing.

The objective is to fix the test `nixosTests.docker-tools`.
2020-05-13 10:29:53 +02:00
Utku Demir f5a90a7aab
dockerTools.buildImage: Preserve environment variables from the parent image 2020-05-08 21:49:16 +12:00
Robert Hensing 6dab1b50a6 buildLayeredImage: Allow empty store, no paths to add
This is useful when buildLayeredImage is called in a generic way
that should allow simple (base) images to be built, which may not
reference any store paths.
2020-02-28 14:59:04 +01:00
Antoine Eiche baa78de594 nixosTests.docker-tools: add bulk-layer test
A regression test for https://github.com/NixOS/nixpkgs/issues/78744.
2020-02-14 09:26:26 +01:00
Richard Wallace 3b65b3f6d6 dockerTools.buildLayeredImage: store all paths passed in final layer
Fixes #78744

My previous change broke when there are more packages than the maximum
number of layers. I had assumed that the `store-path-to-layer.sh` was
only ever passed a single store path, but that is not the case if
there are multiple packages going into the final layer. To fix this, we
loop through the paths going into the final layer, appending them to the
tar file and making sure they end up at the right path.
2020-02-14 09:26:26 +01:00
Antoine Eiche 283bcc1003 dockerTools.buildLayeredImage: fix image with only 2 layers
A test is also added to ensure an image with 2 layers can be built.
2020-01-30 21:09:44 +01:00
Graham Christensen 77452740c6
docker examples: Drop unneeded contents list 2019-12-16 13:11:27 -05:00
Antoine Eiche 09a669478f dockerTools.examples.nix: set USER in the container config
A USER is required by Nix.
See 9348f9291e/src/libutil/util.cc (L478).
2019-12-05 09:45:51 +01:00
Daniël de Kok 3b0d5b5ed3 nixos/tests/docker-tools: check layer sharing with buildLayeredImage
Adapted from grahamc's blog post on layered Docker images in Nix:

https://grahamc.com/blog/nix-and-layered-docker-images
2019-06-06 15:03:08 +02:00
Antoine Eiche 5ef1223f30 nixos/tests/docker-tools: verify order of layers in stacked images 2019-05-07 16:52:13 +02:00
Jaka Hudoklin 468df177c4
dockerTools: add finalImageName parameter for pullImage 2019-03-26 19:35:14 +01:00
Antoine Eiche af23d1e2e7 nixos/test/docker-tools: fix Nix image digest 2019-03-06 09:26:33 +01:00
lewo 7612a6add4
Merge pull request #52870 from xtruder/pkgs/dockerTools/buildLayeredImage/extraCommands
dockerTools: allow to pass extraCommands, uid and gid to buildLayered image
2019-01-10 19:00:19 +01:00
Jaka Hudoklin 954cda5c9d
dockerTools: allow to pass extraCommands, uid and gid to buildLayeredImage 2019-01-10 16:02:23 +01:00
Antoine Eiche 43cbba0274 dockerTools.examples.runAsRootParentImage: init
Example of running something as root on top of a parent image.
This is a regression test related to the PR #52109.
2018-12-27 11:15:35 +01:00
Graham Christensen c88337c9ac
dockerTools.buildImage: support using a layered image in fromImage
Docker images used to be, essentially, a linked list of layers. Each
layer would have a tarball and a json document pointing to its parent,
and the image pointed to the top layer:

    imageA  ----> layerA
                    |
                    v
                  layerB
                    |
                    v
                  layerC

The current image spec changed this format to where the Image defined
the order and set of layers:

    imageA  ---> layerA
            |--> layerB
            `--> layerC

For backwards compatibility, docker produces images which follow both
specs: layers point to parents, and images also point to the entire
list:

    imageA  ---> layerA
            |      |
            |      v
            |--> layerB
            |      |
            |      v
            `--> layerC

This is nice for tooling which supported the older version and never
updated to support the newer format.

Our `buildImage` code only supported the old version, so in order for
`buildImage` to properly generate an image based on another image
with `fromImage`, the parent image's layers must fully support the old
mechanism.

This is not a problem in general, but is a problem with
`buildLayeredImage`.

`buildLayeredImage` creates images with newer image spec, because
individual store paths don't have a guaranteed parent layer. Including
a specific parent ID in the layer's json makes the output less likely
to cache hit when published or pulled.

This means until now, `buildLayeredImage` could not be the input to
`buildImage`.

The changes in this PR change `buildImage` to only use the layer's
manifest when locating parent IDs. This does break buildImage on
extremely old Docker images, though I do wonder how many of these
exist.

This work has been sponsored by Target.
2018-12-05 14:25:54 -05:00
Graham Christensen fb2d153dac
dockerTools: test buildLayeredImage 2018-09-27 14:19:43 -04:00
Graham Christensen aedc651903
dockerTools.buildImage: test that created=now makes an unstable date 2018-09-20 13:06:14 -04:00
Antoine Eiche d44b778d10 dockerTools.examples: explicitly set image tag to fix docker-tools tests
docker-tools tests load images without specifying any tag
value. Docker then uses the image with tag "latest" which doesn't
exist anymore since commit 39e678e24e.
2018-07-06 16:38:42 +02:00
Samuel Dionne-Riel 902b0593be tests/docker-tools: Adds regression test for #34779 2018-05-24 12:23:51 -04:00
Antoine Eiche 736848723e dockerTools.pullImage: Skopeo pulls images by digest
Skopeo is used to pull images from a Docker registry (instead of a
Docker deamon in a VM).

An image reference is specified with its name and its digest which is
an immutable image identifier (unlike image name and tag).

Skopeo can be used to get the digest of an image, for instance:
$ skopeo inspect docker://docker.io/nixos/nix:1.11 | jq -r '.Digest'
2018-05-02 21:32:20 +02:00
Antoine Eiche e8f452f110 dockerTools: add an onTopOfPulledImage example
This allows to test if a pulled image can be updated by using our
Docker tools.
2018-03-13 11:59:22 +01:00
Wei-Ming Yang 70e9b60b33
dockerTools.examples: correct a typo in comments
This commit is for correcting a typo in comments.
2018-01-01 16:13:40 +08:00
Robin Gloster 20677fca59
dockerTools: fix hash to accomodate the pullImage revert 2017-09-28 14:09:49 +02:00
Robin Gloster 5c6dc717a6
Revert "dockerTools.pullImage: use skopeo to pull the image"
This reverts commit 01174c5f4d.

See https://github.com/NixOS/nixpkgs/pull/29302#issuecomment-332809092
for more information. This broke image format compatibility and
therefore amongst others mesos.
2017-09-28 14:09:49 +02:00
Antoine Eiche ff4d7f0fd2 dockerTools.examples.nix: set NIX_PAGER=cat environment variable 2017-09-25 09:39:15 +02:00
Antoine Eiche df589a438e dockerTools.buildImageWithNixDb: populate the Nix Db of the image Nix store
Currently, the contents closure is copied to the layer but there is no
nix database initialization. If pkgs.nix is added in the contents,
nix-store doesn't work because there is no nix database.

From the contents of the layer, this commit generates and loads the
database in the nix store of the container. This only works if there
is no parent layer that already have a nix store (to support several
nix layers, we would have to merge nix databases of parent layers).

We also add an example to play with the nix store inside the
container. Note it seems `more` is a missing dependency of the nix
package!
2017-09-20 20:14:24 +02:00
Antoine Eiche 01174c5f4d dockerTools.pullImage: use skopeo to pull the image
Before this patch, a VM was used to spawn docker that pulled the
VM. Now, the tool Skopeo does this job well so we can simplify our
dockerTools since we doesn't need Docker anymore:)

This also fixe the regression described in
https://github.com/NixOS/nixpkgs/issues/29271 : cntlm proxy doesn't
work in 17.09 while it worked in 17.03.

Note Skopeo doesn't produce the same output than docker pull so, we
have to update sha.
2017-09-17 08:26:02 +01:00
Matej Cotman 8d3cb0af9e docker tools: fix pull image function 2017-05-26 18:48:16 +02:00
Allen Nelson 940aafcbcc use rsync, comment out pulls 2016-10-03 15:12:29 -05:00
Allen Nelson b9007c8d83 add examples file 2016-10-03 12:07:33 -05:00