Commit graph

1478 commits

Author SHA1 Message Date
John Ericson f8ed783f4f meta: Simplify platform check logic
Code golf or readability, you decide
2018-03-27 11:59:59 -04:00
Shea Levy b0482248fe
meta: Add badPlatforms attribute for platform blacklisting. 2018-03-27 08:12:45 -04:00
John Ericson 45e253cacb cross stdenv: Make depsBuildBuild overrideable by config too. 2018-03-20 15:58:09 -04:00
John Ericson 3c331bff5b
Merge pull request #37395 from obsidiansystems/lib-meta-platform
lib: Factor in tiny bit of `meta.platform` checking
2018-03-19 20:12:50 -04:00
John Ericson e547bd0dc4 lib: Factor in tiny bit of meta.platform checking
I need it in stdenv and release-lib, so that seems motivation enough.
2018-03-19 19:29:16 -04:00
John Ericson 2fa2197a96
Merge pull request #34444 from obsidiansystems/meta-check
lib: Fix #30902
2018-03-18 13:51:03 -04:00
John Ericson c26252af3e lib, stdenv: Check meta.platforms against host platform and be open world
First, we need check against the host platform, not the build platform.
That's simple enough.

Second, we move away from exahustive finite case analysis (i.e.
exhaustively listing all platforms the package builds on). That only
work in a closed-world setting, where we know all platforms we might
build one. But with cross compilation, we may be building for arbitrary
platforms, So we need fancier filters. This is the closed world to open
world change.

The solution is instead of having a list of systems (strings in the form
"foo-bar"), we have a list of of systems or "patterns", i.e. attributes
that partially match the output of the parsers in `lib.systems.parse`.
The "check meta" logic treats the systems strings as an exact whitelist
just as before, but treats the patterns as a fuzzy whitelist,
intersecting the actual `hostPlatform` with the pattern and then
checking for equality. (This is done using `matchAttrs`).

The default convenience lists for `meta.platforms` are now changed to be
lists of patterns (usually a single pattern) in
`lib/systems/for-meta.nix` for maximum flexibility under this new
system.

Fixes #30902
2018-03-15 00:44:34 -04:00
John Ericson 4c52e34ca6 stdenv: Clean up check meta args 2018-03-14 18:58:07 -04:00
Jan Malakhovski 7079e744d4 Merge branch 'master' into staging
Resolved the following conflicts (by carefully applying patches from the both
branches since the fork point):

   pkgs/development/libraries/epoxy/default.nix
   pkgs/development/libraries/gtk+/3.x.nix
   pkgs/development/python-modules/asgiref/default.nix
   pkgs/development/python-modules/daphne/default.nix
   pkgs/os-specific/linux/systemd/default.nix
2018-03-10 20:38:13 +00:00
John Ericson 8f54d931cd
Merge pull request #36401 from obsidiansystems/no-stdenv-cross-adapter
stdenv cross adapter: Get rid if `selfNativeBuildInput`
2018-03-06 20:17:13 -05:00
John Ericson 8d58008039 stdenv cross adapter: Get rid if selfNativeBuildInput
It is finally no longer used anywhere

Fixes #30587
2018-03-06 20:05:13 -05:00
Jan Malakhovski 1d4f3fa371 stdenv: check-meta: fix fallout from #36119 2018-03-06 14:14:17 +00:00
Will Dietz ece4c62d4b make-bootstrap-tools: preserve coreutils symlinks
We go out of our way (see top of file) to build a single binary
with symlinks for all of the tools, but were losing them
when preparing the bootstrap tools.
2018-03-06 15:13:56 +02:00
Tuomas Tynkkynen 34f95d92a2 Merge remote-tracking branch 'upstream/master' into staging
Conflicts:
	pkgs/applications/misc/pytrainer/default.nix
	pkgs/development/tools/pew/default.nix
	pkgs/tools/misc/you-get/default.nix
2018-02-28 20:52:49 +02:00
John Ericson 2482e2858e prebuilt android tools: Init using SDK
Expose as an option for the cross stdenv.
2018-02-27 14:15:39 -05:00
John Ericson dfc5d7835d
Merge pull request #35247 from telent/mips32
lib, treewide: Add missing MIPS arches, and fix existing usage
2018-02-27 14:01:15 -05:00
John Ericson 4a29081a94
Merge pull request #35071 from oxij/stdenv/infopages
stdenv, bash: fixing info pages and stuff
2018-02-26 18:06:11 -05:00
Jan Malakhovski 49afe1d490 stdenv: darwin: give distinct names to all the stages 2018-02-26 22:46:34 +00:00
Jan Malakhovski e654d9a9ca stdenv: linux: give distinct names to all the stages
Mainly for debugging.
2018-02-26 22:46:34 +00:00
Jan Malakhovski 522437362a stdenv: linux, darwin: don't build documentation for the the intermediate stages
For the cc of the intermediate stages, to be precise. Doing the same for
bintools requires lots of refactoring.

This is mainly for the future extensibility as now you can change
documentation generation with impunity without rebuilding the
whole of stdenv.
2018-02-26 22:45:10 +00:00
Shea Levy 1c1a6dfd23
libgcrypt: Fix cross-compilation 2018-02-24 22:51:22 -05:00
Daniel Barlow 9c50ae6898 lib, treewide: Add missing MIPS arches, and fix existing usage
Existing "mips64el" should be "mipsel".

This is just the barest minimum so that nixpkgs can recognize them as
systems - although required for building individual derivations onto
MIPS boards, it is not sufficient if you want to actually build nixos on
those targets
2018-02-23 20:43:42 -05:00
Vladimír Čunát 1d15dadbec
Merge branch 'master' into staging
Larger rebuilds from master.
2018-02-20 20:33:40 +01:00
Vladimír Čunát 882062004b
Merge branch 'staging' into gcc-7
I'm betting Hydra resources on gcc-7 getting to master before current staging.
2018-02-19 15:24:32 +01:00
Michael Raskin 7d2642724f
Merge pull request #35111 from oxij/stdenv/available-evaluates
stdenv.mkDerivation: rename `meta.evaluates` -> `meta.available`
2018-02-19 14:15:06 +00:00
Michael Raskin e37b756b91
Merge pull request #35110 from oxij/pkgs/pretty-fetchurl
fetchurl: cleanup, better errors
2018-02-19 14:05:12 +00:00
Shea Levy bbfc9ce376
Merge remote-tracking branch 'origin/master' into staging 2018-02-18 16:56:39 -05:00
Shea Levy 32ce7012f0
Merge branch 'master' into gcc-7 2018-02-18 16:27:22 -05:00
Shea Levy 0a0ecc4c2c
Enable building riscv64 cross bootstrap tools
Fixes #35089
2018-02-18 16:07:13 -05:00
Jan Malakhovski a89899ce4e fetchurl: cleanup, better errors
Also fix what seems like bugs in uncommon `stdenv`s.
2018-02-18 14:24:53 +00:00
Jan Malakhovski 810c4702cf stdenv.mkDerivation: rename meta.evaluates -> meta.available
A much better name.
2018-02-18 13:33:25 +00:00
Shea Levy b24ce2ae63
Handle sourceRoots with leading dashes 2018-02-15 11:50:24 -05:00
Tuomas Tynkkynen b1916b45a3 Merge remote-tracking branch 'upstream/staging' into gcc-7
Conflicts:
	pkgs/development/libraries/libidn/default.nix
	pkgs/top-level/all-packages.nix
2018-02-15 15:45:37 +02:00
Will Dietz 84a527ea5e x86_64-musl bootstrap: use separate 64bit busybox
Avoid issues like #24954.
2018-02-13 09:45:05 -06:00
Will Dietz 4619f8e06d new musl bootstrap URL's, much smaller. rebuild all the things. 2018-02-13 09:45:04 -06:00
Will Dietz 268bff6185 make-bootstrap-tools: set XZ to maximum level, big wins
x86_64 bootstrap tarball goes from 37M -> 21M (!)
2018-02-13 09:45:04 -06:00
Will Dietz a4c69744dc aarch64-musl, armv6l-musl: fetch from wdtz.org instead of storePath 2018-02-13 09:45:04 -06:00
Will Dietz 8bedb690a2 add musl bootstrap paths for aarch64, armv6l
Aarch64 tools tested briefly with qemu-aarch64,
but neither have been actually used yet :).

For now only "host" indirectly via binary cache
at cache.allvm.org.
2018-02-13 09:45:04 -06:00
Will Dietz d00f200f4e bootstrap-files: nicer naming 2018-02-13 09:45:03 -06:00
Will Dietz 119920faa6 restore stdenv.glibc, will remove separately. 2018-02-13 09:45:03 -06:00
Will Dietz c8d03e6298 make-bootstrap-tools: grab libc from stdenv.cc.libc
This is the same in current cases AFAICT,
other than uses musl instead of glibc when musl-native.
2018-02-13 09:44:57 -06:00
Will Dietz 9bc8127dc3 bootstrap-tools-cross, release-cross: add various musl entries 2018-02-13 09:44:56 -06:00
Will Dietz 4b6b83f159 linux stdenv: find bootstrap files by libc, then arch 2018-02-13 09:44:42 -06:00
Will Dietz 5a8002873e fixup linux stdenv bootstrap 2018-02-13 09:44:40 -06:00
Will Dietz 9dbbd75e54 musl bootstrap: allow libiconv 2018-02-13 09:44:40 -06:00
Will Dietz fd00d37b4f musl64 bootstrap: grab from gravity, requires netrc magic 2018-02-13 09:44:39 -06:00
Will Dietz 3ee5094934 linux bootstrap for musl: kludgery 2018-02-13 09:44:39 -06:00
Will Dietz 0e16989d39 bootstrap-tools-musl: WIP 2018-02-13 09:44:37 -06:00
Will Dietz a0af2aadb6 for now, copy bootstrap-tools instead of modify in-place 2018-02-13 09:44:37 -06:00
Will Dietz 0ac504227d make-bootstrap-tools: initial musl support, less glibc-specific 2018-02-13 09:44:37 -06:00
Will Dietz 4831495995 make-bootstrap-tools: glibc -> libcCross 2018-02-13 09:44:37 -06:00
Will Dietz 5dcf3cdf37 stdenv: automatically update config.sub for musl 2018-02-13 09:44:36 -06:00
Will Dietz 5ac5a3ebb5 make-bootstrap-tools-cross.nix: add musl -> musl64 2018-02-13 09:44:35 -06:00
Tuomas Tynkkynen a6fd03876e check-meta: Use concatStrings 2018-02-11 00:17:47 +02:00
Vladimír Čunát 03eef81727
darwin bootstrap tools: fix after #34339 (http2 in curl) 2018-02-10 19:20:44 +01:00
Franz Pletz c135ecd7ef
Merge remote-tracking branch 'origin/master' into gcc-7 2018-02-05 13:11:33 +01:00
Tuomas Tynkkynen 6a11ebfae9 Merge remote-tracking branch 'upstream/master' into gcc-7
Conflicts:
	pkgs/development/compilers/gcc/7/default.nix
	pkgs/development/libraries/SDL2/default.nix
	pkgs/top-level/all-packages.nix
2018-01-26 16:46:43 +02:00
Shea Levy f83b6e1130
unpackPhase: Handle sources starting with a hyphen 2018-01-24 21:58:57 -08:00
John Ericson 5a754e75b4 Merge branch 'ericson2314-cross-master' into staging 2018-01-16 13:05:39 -05:00
Vladimír Čunát 67e8392383
Merge #33057: stdenv meta checks: make them lazy
Closes #22277 - it's superseded;  I have some WIP on evaluation
performance, but best do that in a separate PR/thread.
2018-01-14 21:41:31 +01:00
Tuomas Tynkkynen 6ed0fe7e45 Merge remote-tracking branch 'upstream/master' into staging
Conflicts:
	pkgs/build-support/fetchbower/default.nix
	pkgs/build-support/fetchdarcs/default.nix
	pkgs/build-support/fetchgx/default.nix
	pkgs/development/python-modules/botocore/default.nix
	pkgs/os-specific/linux/firmware/firmware-linux-nonfree/default.nix
	pkgs/tools/admin/awscli/default.nix
2018-01-14 21:18:27 +02:00
Jan Malakhovski fac3d49e48 stdenv: provide meta.name 2018-01-14 13:24:30 +00:00
Jan Malakhovski 50148f0630 stdenv: hide name under check-meta assert
This is a temporary workaround to make `nix-env -qa` and `nix search` ignore
broken packages as they they did before this patchset.

This patch should be reverted after `nix` gets a proper fix for this.
See NixOS/nix#1771.
2018-01-14 13:24:30 +00:00
Jan Malakhovski eaee2a1199 stdenv: implement config.checkMetaRecursively
This option makes `meta.evaluate` into a close approximation of the result of
evaluating `.outPath` by checking all the dependencies recursively at a cost of
2x slowdown. Note that actually evaluating `.outPath` costs some
5x-7x more because `.outPath` also computes all the hashes.
2018-01-14 13:24:30 +00:00
Jan Malakhovski ecd3990cd4 stdenv: provide meta.evaluates
This gives a way to see the result of `check-meta` without triggering any assertions.
2018-01-14 13:24:29 +00:00
Jan Malakhovski 9956687151 stdenv: change some indent 2018-01-14 13:24:22 +00:00
John Ericson e017a027d5
Merge pull request #33681 from obsidiansystems/fixed-output-deps
Fixed output deps
2018-01-10 14:28:10 -05:00
John Ericson 940c4fa3f5 treewide: Fetchers should use stdenvNoCC. 2018-01-10 11:18:44 -05:00
John Ericson c836910e0e
Merge pull request #33670 from obsidiansystems/cross-check
stdenv: Fix doCheck and doInstallCheck logic
2018-01-09 16:37:11 -05:00
John Ericson 567feb3a54 stdenv: Fix doCheck and doInstallCheck logic
It's host != build, not host != target
2018-01-09 16:22:58 -05:00
John Ericson 06a8d66528
Merge pull request #33603 from obsidiansystems/cross-check
stdenv: Force `doCheck` to be false when we are cross compiling
2018-01-09 15:09:54 -05:00
John Ericson 4e907dbca1 stdenv: Force doCheck and doInstallCheck to be false when we are cross compiling
I hope this will be a temporary measure. If there is consensus around
issue #33599, then we can follow an explicit `dontCheck`, but default to
not checking during cross builds when none is given.
2018-01-09 12:37:12 -05:00
Will Dietz 9721ed22e8 schedulingPriority should be an int, fix check-meta type and in-tree use 2018-01-09 07:25:24 -06:00
adisbladis c2316114bc
stdenv: Kill off ensureDir 2018-01-09 11:14:48 +08:00
Daiderd Jordan e66a5f8862
Merge pull request #33342 from LnL7/clang-outputs
clang: add lib output
2018-01-07 09:43:38 +01:00
Jan Malakhovski d1d5ecb3bf stdenv: perform checks only when evaluating .drv and .out
This pushes check-meta evaluation to derivation evaluation step, leaving all other
attributes accessible.

Before this commit:

> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.

as expected

> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.

> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.

which is unfortunate since its impossible to use packages in autogenerated
documentation on all platforms.

After this commit:

> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux

still fails

> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> "xen-4.5.5"

> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> "Xen hypervisor and related components (vanilla)"
2018-01-03 12:20:25 +00:00
Daiderd Jordan b0c043d26c
stdenv: allow lib output of clang on darwin 2018-01-02 20:00:31 +01:00
John Ericson ab651d2c9b linux bootstrap tools: Use same derivation whether cross compiling or not 2018-01-02 13:52:41 -05:00
John Ericson 4d2b763817
Merge pull request #26805 from obsidiansystems/cross-elegant
Make cross compilation elegant
2017-12-30 22:58:02 -05:00
John Ericson 469fd89832 stdenv-setup: Ease the transition with native builds
- All deps go on the PATH

 - CC and Bintools wrappers with their host != depender's host still get their
   setup hooks run.

 - Environment hooks get applied to all packages

This isn't so elegent, but eases the transition on a very significant
PR.
2017-12-30 22:04:23 -05:00
John Ericson f083248290 linux bootstrap tools cross: Nuke more refs
libgcc.a and similar
2017-12-30 22:04:23 -05:00
John Ericson a036473a0a {bintools,cc}-wrapper: Fix setup hook to respect the role of the cc-compiler
We now have the information to properly determine the role the
cc-wrapper dependency has, by taking advantage of `offset`. No longer
use the soon-to-be-deprecated crossConfig environment variable, the
temp hack used before this change.
2017-12-30 22:04:21 -05:00
John Ericson 7f3ca3e21a stdenv: Fix handling of dependencies and hooks
4 far-reaching changes: Smaller PATH, New vars, different propagation
logic, and different hook logic

Smaller PATH
------------

`buildInputs` no longer go on the PATH at build time, as they cannot be
run when cross compiling and we don't want to special case. Simply make
a `nativeBuildInput` too if one needs them on the PATH. Fixes #21191.

Many new depedendency variables
-------------------------------

See the stdenv chapter of the nixpkgs manual. I pulled out the existing
documentation of dependency specification into a new section, and added
language for these two (and their propagated equivalents) along side
the others'.

More complex propagation logic
------------------------------

Before a propagated*XXX*Input always acted as if it was specified
directly as a *XXX*Input downstream. That's simple enough, but violates
the intended roles of each sort of dep, which has functional and not
just stylistic consequences.

The new algorithm is detailed in the manual, and ensures everything
ends up in the right place. I tried to give both an informal and formal
description, but I suspect in practice it will not make much sense
until one tries cross compiling, after which it will immediately make
sense as the only sane option.

Simplified hook logic
---------------------

Rather than `envHook` and `crossEnvHook`, whose behavior differs
depending on whether we are cross compiling or not, there is now one
hook per sort (or rather non-propagated and propagated pair of sorts)
of dependency. These new hooks have the same meaning regardless of
cross compilation. See the setup hook section of stdenv chapter of the
Nixpkgs manual for more details.
2017-12-30 22:04:21 -05:00
John Ericson eb27be0731
Merge pull request #33186 from obsidiansystems/cross-binutils
binutils: Fix cross, again
2017-12-29 17:42:38 -05:00
John Ericson 1cde06acf6 stdenv cross adapter: Get rid of extra propagatation of buildInputs
Binutils is patched so we don't rely on `--rpath-link`.
2017-12-29 17:32:28 -05:00
Tuomas Tynkkynen 47c782f7f5 Merge remote-tracking branch 'upstream/master' into staging 2017-12-26 21:10:12 +02:00
John Ericson 4651407654 darwin stdenv: Make stdenv.cc, not stdenv, bring in libcxx
stdenvNoCC should not inject any C++ standard library, just as it
doesn't inject any C standard library. stdenv still does, but only
indirectly through stdenv.cc. Wrapped clangs can be simplified now that
they don't need to worry about clobbering CoreFoundation when replacing
the C++ standard library implementation.

This generally-good cleanup should assist with debugging some C++
failures in #26805.
2017-12-25 19:32:07 -05:00
Vladimír Čunát ced4e5a683
darwin stdenv boostrap tools: use curl without kerberos
/cc #29785.  Otherwise we would have to put the lib in, etc.
2017-12-24 11:10:10 +01:00
Vladimír Čunát 24d81d6332
Merge branch 'master' into staging 2017-12-15 21:40:23 +01:00
Graham Christensen ae9a7c4969
stdenv: make knownVulnerabilities a known meta-type 2017-12-14 07:36:31 -05:00
John Ericson a0b1ebeee9 Merge remote-tracking branch 'upstream/staging' into binutils-wrapper 2017-12-13 16:14:47 -05:00
John Ericson 2bba929062 bintools-wrapper: Import separately from cc-wrapper 2017-12-13 16:08:18 -05:00
Graham Christensen f33a513d2b
stdenv: allow specifying a eval issuee handler 2017-12-12 18:08:11 -05:00
Vladimír Čunát 76bf375a16
stdenv checkMeta: throw -> trace
- tracing seems annoying enough
- we get errors for all packages instead of aborting on the first one
- easier to differentiate from unwanted packages (broken, unfree, etc.)
2017-12-12 18:07:07 -05:00
Will Dietz db7bee240e stdenv: fix use of config-specified allowInsecurePredicate
for some reason we were checking allowUnfreePredicate instead
2017-12-09 20:43:30 +00:00
Jan Malakhovski 1858e8909e tree-wide: fix either check-meta or meta attrs of all the packages I evaluate 2017-12-05 13:46:52 +01:00
Jan Malakhovski 8ae51ff9c1 stdenv/generic/check-meta: fix error message evaluation 2017-12-05 13:46:52 +01:00
John Ericson 405412dfd9
Merge pull request #31775 from obsidiansystems/stdenv-both-propagated-files
stdenv setup: Always use both propagated files
2017-11-22 15:23:37 -05:00
John Ericson da19c34d0f stdenv setup: Always use both propagated files
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.

This is an incremental step to fully keeping the dependencies separate
in all cases.
2017-11-21 10:44:44 -05:00
Orivej Desh d99a2fc093 Merge branch 'master' into staging
* master: (293 commits)
  go_1_9: skip flaky TestServerCancelsReadTimeoutWhenIdle
  qsyncthingtray: fix build
  qt56.qtwebengine: fix build
  stdman: d860212 -> 2017.04.02
  jackett: use mono50
  hg-git: disable with python3
  hg-git: 0.8.5 -> 0.8.10
  xfce4-settings: enable parallel building
  gcc-snapshot: mark as broken
  heaptrack: 2017-02-14 -> 2017-10-30
  nixos-container: Modify existing test to cover show-ip command
  nixos-container: Make show-ip work together with ipv4 + netmask
  linux-copperhead: 4.13.12.a -> 4.13.13.a
  matterbridge: 1.1.0 -> 1.4.1
  nixos/nghttpx: add module for the nghttpx proxy server (#31680)
  mattermost: 4.3.0 -> 4.4.0
  breakpad: delete
  simp_le: 0.2.0 -> 0.6.1
  certbot: 0.11.1 -> 0.19.0
  afl: 2.51b -> 2.52b
  ...
2017-11-17 05:35:09 +00:00