Commit graph

47 commits

Author SHA1 Message Date
talyz b7749c7671
nixos/test-driver: Run commands with error handling
Bash's standard behavior of not propagating non-zero exit codes
through a pipeline is unexpected and almost universally
unwanted. Default to setting `pipefail` for the command being run;
it can still be turned off by prefixing the pipeline with
`set +o pipefail` if needed.

Also, set `errexit` and `nonunset` options to make the first command
of consecutive commands separated by `;` fail, and disallow
dereferencing unset variables respectively.
2021-06-05 18:44:42 +02:00
Robert Hensing 5259d66b74 dockerTools: Allow omitting all store paths
Adds includeStorePaths, allowing the omission of the store paths.
You generally want to leave it on, but tooling may disable this
to insert the store paths more efficiently via other means, such
as bind mounting the host store.
2021-05-26 15:11:42 +02:00
Robert Hensing 8c868f47a8 Revert "nixos/tests/docker-tools*: remove useless formatter"
Annoyed with the interference of the python formatting of
generated code (see #72964), I took matters into my own hands
as maintainer of dockerTools.

Afterwards, I've created a PR, hoping to unstuck the discussion.

@aszlig took notice and thanks to his python ecosystem knowledge,
the testing efforts of @blaggacao and @Ma27, and a sense of
shared suffering and comraderie we were able to change the
situation for the better in #122201.

Now, we have a proper linter that actually helps contributors,
so it's time to turn it back on again.

I'm glad we could make it happen this quickly!

Thanks!

This reverts commit 4035049af3.
2021-05-09 02:57:17 +02:00
Robert Hensing 4035049af3 nixos/tests/docker-tools*: remove useless formatter 2021-05-08 15:03:20 +02:00
Robert Hensing a67c97a5eb nixos/tests/docker-tools*: Add myself as maintainer where missing
I should have done this when I became maintainer for dockerTools,
but it's the PR reviews that matter.
2021-05-08 15:00:19 +02:00
Viktor Kronvall 10019ab879 dockerTools: test that mergeImages preserves owner 2021-04-08 00:22:59 +09:00
Viktor Kronvall bcc3f8eab5 dockerTools: test mergeImages 2021-04-08 00:22:59 +09:00
Robert Hensing 58b21dea78
Merge pull request #116749 from vroad/docker-layered-image-fakeroot
dockerTools.streamLayeredImage: add fakeRootCommands option
2021-04-07 15:02:24 +02:00
vroad 63e7c4186f dockerTools.streamLayeredImage: Add test for fakeRootCommands 2021-04-07 18:11:02 +09:00
Louis Blin b3f68289df dockerTools.streamLayeredImage: resolve duplicate env vars
For images running on Kubernetes, there is no guarantee on how duplicate
environment variables in the image config will be handled. This seems
to be different from Docker, where the last environment variable value
is consistently selected.

The current code for `streamLayeredImage` was exploiting that assumption
to easily propagate environment variables from the base image, leaving
duplicates unchecked. It should rather resolve these duplicates to
ensure consistent behavior on Docker and Kubernetes.
2021-03-25 23:29:54 +00:00
Louis Blin aae8588182 dockerTools.buildLayeredImage: support fromImage
It is now possible to pass a `fromImage` to `buildLayeredImage` and
`streamLayeredImage`, similar to what `buildImage` currently supports.

This will prepend the layers of the given base image to the resulting
image, while ensuring that at most `maxLayers` are used. It will also
ensure that environment variables from the base image are propagated
to the final image.
2021-03-23 14:50:42 +00:00
Louis Blin 419a4fa596 dockerTools.buildLayeredImage: image names with registry/ prefix
When using `buildLayeredImage`, it is not possible to specify an image
name of the form `<registry>/my/image`, although it is a valid name.

This is due to derivations under `buildLayeredImage` using that image
name as their derivation name, but slashes are not permitted in that
context.

A while ago, #13099 fixed that exact same problem in `buildImage` by
using `baseNameOf name` in derivation names instead of `name`. This
change does the same thing for `buildLayeredImage`.
2021-03-10 17:44:24 +00:00
Mark Vainomaa f53cf9a65c
Revert "nixos/tests/docker: wait for docker service"
This reverts commit c14bafd6c7db3b0c03574e28831821adda22a6f1.
2021-01-15 15:53:43 +02:00
Milan f634c149e4
nixos/tests/docker: wait for docker service (#109418)
Previously the Docker daemon was started by systemd socket activation.
Thus, the Docker test waited for the sockets.target unit.
But when the docker module was changed to start the Docker daemon at
boot instead of by socket activation, the test was left untouched.
With the Docker 20.10 update this lead to a timing issue, where the
docker command is run before the Docker daemon has started and hangs.

Fixes #109416
2021-01-15 03:38:15 +01:00
Dominik Xaver Hörl 25bef2d8f9 treewide: simplify pkgs.stdenv.lib -> pkgs.lib
The library does not depend on stdenv, that `stdenv` exposes `lib` is
an artifact of the ancient origins of nixpkgs.
2021-01-10 20:12:06 +01:00
Sarah Brofeldt ffe5ff6009 dockerTools: Test buildLayeredImage with symlinks
This exercises layer creation in face of store path symlinks, ensuring
they are not dereferenced, which can lead to broken layer tarballs
2021-01-04 21:44:47 +01:00
Terin Stock 8f66dc94a7 dockerTools: normalize arch to GOARCH
Docker (via containerd) and the the OCI Image Configuration imply and
suggest, respectfully, that the architecture set in images matches those
of GOARCH in the Go Language document.

This changeset updates the implimentation of getArch in dockerTools to
return GOARCH values, to satisfy Docker.

Fixes: #106695
2020-12-15 02:14:01 -08:00
Sarah Brofeldt 67650f71cd nixos/tests/docker-tools.nix: Fix format 2020-12-02 08:03:38 +01:00
adisbladis b7b22c5814
dockerTools: Always cross compile for another arch in the cross example
The example fails to build on aarch64, so lets cross build for gnu64.
2020-11-20 12:57:58 +01:00
adisbladis 11367b2db1
dockerTools: Add cross compilation test 2020-11-19 18:13:22 +01:00
Scott Worley c25ccf6b4b nixos/tests/docker-tools: Use curl --fail 2020-10-25 11:01:31 +01:00
Utku Demir ae82f81bfa
dockerTools.streamLayeredImage: Store the customisation layer as a tarball
This fixes as issue described here[1], where permissions set by 'extraCommands'
were ignored by Nix.

[1] https://github.com/NixOS/nixpkgs/pull/91084#issuecomment-669834938
2020-09-04 16:53:23 +12:00
Johan Thomsen f5db415e2f nixos/tests/dockerTools: add test for running non-root containers with buildLayeredImage
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-07-31 10:14:07 +02:00
Robert Hensing 8c0459f611 dockerTools: Always set imageTag attribute
The image tag can be specified or generated from the output hash.
Previously, a generated tag could be recovered from the evaluated
image with some string operations.

However, with the introduction of streamLayeredImage, it's not
feasible to compute the generated tag yourself.

With this change, the imageTag attribute is set unconditionally,
for the buildImage, buildLayeredImage, streamLayeredImage functions.
2020-07-11 16:58:25 +02:00
Jos van Bakel 66737168a4
dockerTools.buildLayeredImage: fix created=now 2020-07-09 09:34:18 +02:00
Utku Demir 06db331922
dockerTools: Verify nix-store contents on buildLayeredImage test 2020-07-06 16:59:58 +12:00
Utku Demir cc46362929
dockerTools: Support files directly under /nix/store
Also makes sure that the files inside a layer added in a sorted order
to make the results more deterministic.
2020-07-04 22:00:57 +12:00
Robert Hensing caf47063b4 dockerTools: test that tar keeps nix binary symlinks intact 2020-06-19 10:08:34 +02:00
Utku Demir f5a90a7aab
dockerTools.buildImage: Preserve environment variables from the parent image 2020-05-08 21:49:16 +12:00
Robert Hensing 6dab1b50a6 buildLayeredImage: Allow empty store, no paths to add
This is useful when buildLayeredImage is called in a generic way
that should allow simple (base) images to be built, which may not
reference any store paths.
2020-02-28 14:59:04 +01:00
Jacek Galowicz 0a49699499 nixosTests.docker-tools: Port to Python 2020-02-27 17:15:31 +01:00
Antoine Eiche baa78de594 nixosTests.docker-tools: add bulk-layer test
A regression test for https://github.com/NixOS/nixpkgs/issues/78744.
2020-02-14 09:26:26 +01:00
Antoine Eiche 283bcc1003 dockerTools.buildLayeredImage: fix image with only 2 layers
A test is also added to ensure an image with 2 layers can be built.
2020-01-30 21:09:44 +01:00
Daniël de Kok 3b0d5b5ed3 nixos/tests/docker-tools: check layer sharing with buildLayeredImage
Adapted from grahamc's blog post on layered Docker images in Nix:

https://grahamc.com/blog/nix-and-layered-docker-images
2019-06-06 15:03:08 +02:00
Antoine Eiche 5ef1223f30 nixos/tests/docker-tools: verify order of layers in stacked images 2019-05-07 16:52:13 +02:00
Jaka Hudoklin 468df177c4
dockerTools: add finalImageName parameter for pullImage 2019-03-26 19:35:14 +01:00
Antoine Eiche af23d1e2e7 nixos/test/docker-tools: fix Nix image digest 2019-03-06 09:26:33 +01:00
Jaka Hudoklin 954cda5c9d
dockerTools: allow to pass extraCommands, uid and gid to buildLayeredImage 2019-01-10 16:02:23 +01:00
Graham Christensen c88337c9ac
dockerTools.buildImage: support using a layered image in fromImage
Docker images used to be, essentially, a linked list of layers. Each
layer would have a tarball and a json document pointing to its parent,
and the image pointed to the top layer:

    imageA  ----> layerA
                    |
                    v
                  layerB
                    |
                    v
                  layerC

The current image spec changed this format to where the Image defined
the order and set of layers:

    imageA  ---> layerA
            |--> layerB
            `--> layerC

For backwards compatibility, docker produces images which follow both
specs: layers point to parents, and images also point to the entire
list:

    imageA  ---> layerA
            |      |
            |      v
            |--> layerB
            |      |
            |      v
            `--> layerC

This is nice for tooling which supported the older version and never
updated to support the newer format.

Our `buildImage` code only supported the old version, so in order for
`buildImage` to properly generate an image based on another image
with `fromImage`, the parent image's layers must fully support the old
mechanism.

This is not a problem in general, but is a problem with
`buildLayeredImage`.

`buildLayeredImage` creates images with newer image spec, because
individual store paths don't have a guaranteed parent layer. Including
a specific parent ID in the layer's json makes the output less likely
to cache hit when published or pulled.

This means until now, `buildLayeredImage` could not be the input to
`buildImage`.

The changes in this PR change `buildImage` to only use the layer's
manifest when locating parent IDs. This does break buildImage on
extremely old Docker images, though I do wonder how many of these
exist.

This work has been sponsored by Target.
2018-12-05 14:25:54 -05:00
Graham Christensen fb2d153dac
dockerTools: test buildLayeredImage 2018-09-27 14:19:43 -04:00
Graham Christensen aedc651903
dockerTools.buildImage: test that created=now makes an unstable date 2018-09-20 13:06:14 -04:00
volth 2e979e8ceb [bot] nixos/*: remove unused arguments in lambdas 2018-07-20 20:56:59 +00:00
Samuel Dionne-Riel 902b0593be tests/docker-tools: Adds regression test for #34779 2018-05-24 12:23:51 -04:00
Daiderd Jordan d538fc06e2
docker-tools: add a test for permissions issues with AUFS/overlay
docker# [   11.054736] d24d6cdd57c9[763]: /bin/bash: error while loading
shared libraries: libreadline.so.7: cannot open shared object file:
Permission denied
docker# /bin/bash: error while loading shared libraries:
libreadline.so.7: cannot open shared object file: Permission denied
docker: exit status 127
docker: output:
error: command `docker run --rm -u 1000:1000 bash /bin/bash --version'
did not succeed (exit code 127)
command `docker run --rm -u 1000:1000 bash /bin/bash --version' did not
succeed (exit code 127)
2018-04-14 14:11:14 +02:00
Antoine Eiche bc1e8f95d4 tests/docker-tools: add onTopOfPulledImage test 2018-03-13 11:59:22 +01:00
Antoine Eiche f9bb73f4aa nixos.test.dockerTools: add more tests 2018-02-16 18:24:48 +01:00
Profpatsch 1545f9062e nixos/tests: add simple dockerTools test 2018-02-14 06:36:14 +01:00