Commit graph

3033 commits

Author SHA1 Message Date
lewo f7e67be1dc
Merge pull request #51528 from grahamc/buildImage-on-layered-image
dockertools buildImage: support new-style image specs
2018-12-07 09:44:58 +01:00
Graham Christensen c88337c9ac
dockerTools.buildImage: support using a layered image in fromImage
Docker images used to be, essentially, a linked list of layers. Each
layer would have a tarball and a json document pointing to its parent,
and the image pointed to the top layer:

    imageA  ----> layerA
                    |
                    v
                  layerB
                    |
                    v
                  layerC

The current image spec changed this format to where the Image defined
the order and set of layers:

    imageA  ---> layerA
            |--> layerB
            `--> layerC

For backwards compatibility, docker produces images which follow both
specs: layers point to parents, and images also point to the entire
list:

    imageA  ---> layerA
            |      |
            |      v
            |--> layerB
            |      |
            |      v
            `--> layerC

This is nice for tooling which supported the older version and never
updated to support the newer format.

Our `buildImage` code only supported the old version, so in order for
`buildImage` to properly generate an image based on another image
with `fromImage`, the parent image's layers must fully support the old
mechanism.

This is not a problem in general, but is a problem with
`buildLayeredImage`.

`buildLayeredImage` creates images with newer image spec, because
individual store paths don't have a guaranteed parent layer. Including
a specific parent ID in the layer's json makes the output less likely
to cache hit when published or pulled.

This means until now, `buildLayeredImage` could not be the input to
`buildImage`.

The changes in this PR change `buildImage` to only use the layer's
manifest when locating parent IDs. This does break buildImage on
extremely old Docker images, though I do wonder how many of these
exist.

This work has been sponsored by Target.
2018-12-05 14:25:54 -05:00
Austin Seipp 3aa9091162 buildRustCreate: export RUSTDOC during cargo config
See https://github.com/NixOS/nixpkgs/pull/50452#issuecomment-443455411

Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-12-04 19:51:51 -06:00
Jörg Thalheim 8c74bea7b8
Merge pull request #51353 from krebs/nix-writers
build-support writers: content can be string or file
2018-12-04 20:13:36 +00:00
lassulus 995defbb9b build-support writers: content can be string or file 2018-12-04 16:40:32 +01:00
Graham Christensen fc459de60e
Merge pull request #50452 from P-E-Meunier/carnix-fix
Rust build-support: fixing a compilation error in some crates
2018-12-03 10:53:25 -05:00
Frederik Rietdijk 1828a5c5ba Merge master into staging-next 2018-11-30 17:46:21 +01:00
Vincent Laporte 66efb76c75 ocamlPackages.buildDunePackage: fix meta.platforms 2018-11-30 16:19:03 +00:00
Charles Duffy 8f90b33240
buildMaven: Support metadata-only dependencies
The circumstances which can make this necessary are discussed in NixOS/mvn2nix-maven-plugin#17
2018-11-29 17:29:59 +00:00
Frederik Rietdijk 9db2421d1f Merge master into staging-next 2018-11-29 08:12:56 +01:00
Matthew Bauer 7fa8c41c4a
Merge pull request #50235 from illegalprime/more-fhsuserenv-blacklists
fix FHSUserEnv blacklists
2018-11-28 09:32:01 -06:00
Pierre-Étienne Meunier 3083fa2aa1 Carnix 0.9.2 2018-11-27 16:08:11 +00:00
Thomas Tuegel fb7749620b
melpaBuild: Get package-build from melpa/package-build
This commit causes melpaBuild to use package-build from melpa/package-build
instead of melpa/melpa. Development of package-build happens in the former
repository whereas the latter is much larger, containing also the MELPA
recipes. We do not need to fetch the MELPA recipes from melpa/melpa, as we fetch
them one-by-one for Nixpkgs.
2018-11-27 09:36:28 -06:00
Thomas Tuegel 762295a39c
Merge pull request #51095 from bhipple/fix/emacs-builders
emacsPackagesNg.trivialBuild: cleanup and standardize function
2018-11-27 09:34:12 -06:00
Jörg Thalheim afbdeb7b9b
Merge pull request #50802 from aszlig/autopatchelf-improvements
autoPatchelfHook: Fixes/improvements for Android SDK emulator
2018-11-27 10:25:26 +00:00
Jörg Thalheim f12bd000b9
Merge pull request #49290 from krebs/nix-writers
get nix-writers into nixpkgs
2018-11-27 07:17:03 +00:00
lassulus 2d02cd7790 build-support writers: add tests 2018-11-27 07:08:13 +01:00
lassulus abd0efae35 build-support: add writers from krebs/writers
Reference https://github.com/krebs/nix-writers revision 40fde9e
2018-11-27 07:08:12 +01:00
Benjamin Hipple a7d1474023 emacsPackagesNg.trivialBuild: cleanup and standardize function
No real function change here, but this updates the trivial and melpa builders to
be formatted more consistently with the rest of the builders, and swaps
`eval "$preBuild"` for the more standard `runHook preBuild`.
2018-11-27 00:23:04 -05:00
aszlig 4a6e3e4185
autoPatchelfHook: Skip on missing segment headers
If the file in question is not a shared object file but an ELF, we
really want to skip the file, because we won't have anything to patch
there.

For example if the file is created via "gcc -c -o foo.o foo.c", we don't
get a segment header and so far autoPatchelf was trying to patch such a
file.

By checking for missing segment headers, we're now no longer going to
attempt patching such a file.

Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: Sander van der Burg <svanderburg@gmail.com>
2018-11-26 01:58:36 +01:00
aszlig 9f23a63f79
autoPatchelfHook: Fix type of norecurse variable
While declaring it as an array doesn't do any harm in our usage, it
might be a bit confusing when reading the code.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-26 01:13:59 +01:00
aszlig 2faf905f98
autoPatchelfHook: Add addAutoPatchelfSearchPath
This function is useful if autoPatchelf is invoked during some of the
phases of a build and allows to add arbitrary shared objects to the
search path.

So far the same functionality was in autoPatchelf itself, but not
available as a separate function, so when adding shared objects to the
dependency cache one would have to do so manually.

The function also has the --no-recurse flag, which prevents recursing
into subdirectories.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-25 16:22:32 +01:00
Frederik Rietdijk e41154d1ab Merge master into staging-next 2018-11-24 10:43:33 +01:00
Jörg Thalheim c424c1161b
Merge pull request #50560 from oxij/pkgs/fetchurl-mirrors
fetchurl: mirrors: http -> https, https before http, http before ftp
2018-11-23 11:56:44 +00:00
Pierre-Étienne Meunier 0e8332ca2b Fixing "include" 2018-11-22 11:40:03 +00:00
Jan Malakhovski 8dcf831978 fetchurl: mirrors: http -> https, https before http, http before ftp
Because HTTP has a higher probability of working behind proxies.
2018-11-22 09:38:22 +00:00
Frederik Rietdijk c31cb577ae Merge master into staging-next 2018-11-22 09:57:08 +01:00
Matthew Bauer 4e68511bb1 bintools: use i386 on all 32 bit x86 systems 2018-11-21 09:38:28 -06:00
Jörg Thalheim 952f4fda86
makeRustPlatform: refactor to make it easier to understand
It is now clearer what is supposed to be in the rust attribute set
without having studied type theory. The amount of code is identically.
2018-11-21 12:44:58 +00:00
aszlig 3ca35ce0b2
autoPatchelfHook: Add --no-recurse flag
This is to be used with the autoPatchelf command and allows to only
patch a specific file or directory without recursing into
subdirectories.

Apart from being able to run the command in a standalone way, as
detailled in the previous commit this is also needed for the Android SDK
emulator, because according to @svanderburg there are subdirectories we
don't want to patch.

The reason why I didn't use GNU getopt is that it might not be available
on all operating systems and the getopts bash builtin doesn't support
long arguments. Apart from that, the implementation for recognizing the
flag is pretty trivial and it's also using bash builtins only, so if we
want to do something really fancy someday, we can still change it.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-20 00:11:29 +01:00
aszlig e4fbb244ee
autoPatchelfHook: Allow to prevent automatic run
If you want to only run autoPatchelf on a specific path and leave
everything else alone, we now have a $dontAutoPatchelf environment
variable, which causes the postFixup hook to not run at all.

The name "dontAutoPatchelf" probably is a bit weird in conjunction with
putting "autoPatchelfHook" in nativeBuildInputs, but unless someone
comes up with a better name I keep it that way because it's consistent
with all the other dontStrip, dontPatchShebangs, dontPatchELF and
whatnot.

A specific example where this is needed is when building the Android SDK
emulator, which contains a few ARM binaries in subdirectories that
should not be patched. If we were to run autoPatchelf on all outputs
unconditionally we'd run into errors because some ARM libraries couldn't
be found.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-20 00:07:38 +01:00
aszlig d03e4ffdbf
autoPatchelfHook: Make easier to run autoPatchelf
The autoPatchelf main function which is run against all of the outputs
was pretty much tailored towards this specific setup-hook and was
relying on $prefix to be set globally.

So if you wanted to run autoPatchelf manually - let's say during
buildPhase - you would have needed to run it like this:

  prefix=/some/directory autoPatchelf

This is now more intuitive and all you need to do is run the following:

  autoPatchelf /some/directory

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-19 17:18:27 +01:00
Frederik Rietdijk 0d0d7dcd06 Merge staging-next into staging 2018-11-18 10:41:34 +01:00
Austin Seipp 04a543b3a0 defaultCrateOverrides: foundationdb native dependencies
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-17 19:28:48 -06:00
zimbatm 551aecfa83
tmpdir audit: only fail with files referenced below (#35068)
On Linux the `$TMPDIR` is `/build`. The TMPDIR audit looks for `$TMPDIR`
in the build output, which will then fail with packages like
/buildkite-agent.

This fixes the heuristic to look for `$TMPDIR/` instead.
2018-11-16 22:35:56 +01:00
Pierre-Étienne Meunier f1de24feb8 Rust build-support: fixing a compilation error in some crates (such as proc-macro2) 2018-11-16 12:12:59 +00:00
Daiderd Jordan bdec3ed049
Revert "Revert "Revert "patch-shebangs: respect cross compilation"""
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.

This reverts commit bde99096a8.
2018-11-14 23:37:32 +01:00
Daiderd Jordan c9223a17bc
Revert "patch-shebangs: use --build for auto patch shebangs"
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.

This reverts commit eb7c50a993.
2018-11-14 23:37:31 +01:00
zimbatm 695a3d4254
Merge pull request #50302 from zimbatm/libredirect-misc
libredirect: misc changes
2018-11-14 00:16:56 +01:00
zimbatm d04a1265a1
libredirect: set install_name on Darwin
fixes https://github.com/NixOS/nixpkgs/pull/50246#issuecomment-437975038
2018-11-14 00:05:26 +01:00
zimbatm 91c130e2f5
libredirect: introduce optional setup-hook
This allows to simplify the usage of libredirect inside of nix build
sandboxes. Add "libredirect.hook" to the build inputs to get everything
linked in automaticall. All that's left is to set NIX_REDIRECTS and call
the target program.
2018-11-14 00:05:23 +01:00
Matthew Bauer 4b8c1d23d0
Merge pull request #42794 from telent/make-closure-x
make-closure: needs build system mkdir and jq
2018-11-13 15:02:08 -06:00
Matthew Bauer f9a6963d9a
Merge pull request #50244 from tathougies/travis/wrap-correctly
make-wrapper should use runtimeShell, not $SHELL, for cross-compilation
2018-11-13 13:55:26 -06:00
zimbatm e62db105c4
libredirect: specify libName
reduces a bit of duplication and can also be used from the outside:

   export LD_PRELOAD=${libredirect}/lib/${libredirect.libName}
2018-11-13 12:26:15 +01:00
Frederik Rietdijk 3b052406ea Merge staging-next into staging 2018-11-12 19:01:36 +01:00
Antoine Eiche c12f75649e dockerTools.buildImageWithNixDb: simplifications and switch to closureInfo
Since Nix 2 is now the stable Nix version, we can use closureInfo
which simplifies the Nix database initialisation (size and hash are
included in the "dump").
2018-11-12 18:30:53 +01:00
aszlig a815f53c60
libredirect: Add preload wrapper for stat()
Pull request #50246 was merged a bit too quickly and it was supposed to
fix libredirect on Darwin. However it still failed on Darwin and this
was missed by the person merging the pull request.

The reason this was failing was that there is no __xstat* on Darwin.

So I'm adding a wrapper for stat() as well as it works on Darwin and it
still doesn't hurt on GNU/Linux.

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edolstra, @zimbatm
2018-11-12 13:31:43 +01:00
aszlig 34dd1c68f8
libredirect: Add a small test
This is just a sanity check on whether the library correctly wraps the
syscalls and it's using the "true" executable for posix_spawn() and
execv().

The installCheckPhase is not executed if we are cross-compiling, so this
shouldn't break cross-compilation.

One thing I'm not actually sure is whether ${coreutils}/bin/true is
universally available on all the platforms, nor whether all the
functions we use in the test are available, but we can still fix that
after we've found out about that.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-12 11:02:54 +01:00
aszlig ba1fddb315
libredirect: Use extensions.sharedLibrary
This is to make sure we get the correct shared library suffix of the
target platform. While for example on Darwin it would even work with the
hardcoded .so prefix it's IMHO a bit nicer to have the actual native
extension.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-12 10:08:02 +01:00
zimbatm 9ef52352bd
assume that it works on all unix platforms 2018-11-12 00:09:36 +01:00