Commit graph

706 commits

Author SHA1 Message Date
Matthew Bauer 9abff4af4f wasm: init cross target
Adds pkgsCross.wasm32 and pkgsCross.wasm64. Use it to build Nixpkgs
with a WebAssembly toolchain.

stdenv/cross: use static overlay on isWasm

isWasm doesn’t make sense dynamically linked.
2019-04-23 21:48:57 -04:00
Graham Christensen 817c933878
check-env: don't execute check-meta.nix 15,000 times
Generated from https://github.com/NixOS/nix/pull/2761:

```
                                                                                          ns     calls ns/call
- /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 591200 15026 39.3451
+ /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 8744   308   28.3896
```

more, generated by:

```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > before 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' before | sort  > before.list
```

applying this patch, then:

```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > after 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' after | sort  > after.list
```

and then diffing before.list and after.list to get:

```
                                                                                                        calls
- :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1               7513
+ :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1               154

- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23                7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23                154

- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26                7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26                154

- onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 15026
+ onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 308
```

The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true`:

| stat                       | before         | after          | Δ               | Δ%      |
|:---------------------------|---------------:|---------------:|:----------------|--------:|
| **cpuTime**                |        179.915 |        145.543 | 🡖 34.372        | -19.10% |
| **envs-bytes**             |  3,900,878,824 |  3,599,483,208 | 🡖 301,395,616   |  -7.73% |
| **envs-elements**          |    214,426,071 |    185,881,709 | 🡖 28,544,362    | -13.31% |
| **envs-number**            |    136,591,891 |    132,026,846 | 🡖 4,565,045     |  -3.34% |
| **gc-heapSize**            | 11,400,048,640 | 12,314,890,240 | 🡕 914,841,600   |   8.02% |
| **gc-totalBytes**          | 25,976,902,560 | 24,510,740,176 | 🡖 1,466,162,384 |  -5.64% |
| **list-bytes**             |  1,665,290,080 |  1,665,290,080 | 0               |         |
| **list-concats**           |      7,264,417 |      7,264,417 | 0               |         |
| **list-elements**          |    208,161,260 |    208,161,260 | 0               |         |
| **nrAvoided**              |    191,359,386 |    179,693,661 | 🡖 11,665,725    |  -6.10% |
| **nrFunctionCalls**        |    119,665,062 |    116,348,547 | 🡖 3,316,515     |  -2.77% |
| **nrLookups**              |     80,996,257 |     76,069,825 | 🡖 4,926,432     |  -6.08% |
| **nrOpUpdateValuesCopied** |    213,930,649 |    213,930,649 | 0               |         |
| **nrOpUpdates**            |     12,025,937 |     12,025,937 | 0               |         |
| **nrPrimOpCalls**          |     88,105,604 |     86,451,598 | 🡖 1,654,006     |  -1.88% |
| **nrThunks**               |    196,842,044 |    175,126,701 | 🡖 21,715,343    | -11.03% |
| **sets-bytes**             |  7,678,425,776 |  7,285,767,928 | 🡖 392,657,848   |  -5.11% |
| **sets-elements**          |    310,241,340 |    294,373,227 | 🡖 15,868,113    |  -5.11% |
| **sets-number**            |     29,079,202 |     27,601,310 | 🡖 1,477,892     |  -5.08% |
| **sizes-Attr**             |             24 |             24 | 0               |         |
| **sizes-Bindings**         |              8 |              8 | 0               |         |
| **sizes-Env**              |             16 |             16 | 0               |         |
| **sizes-Value**            |             24 |             24 | 0               |         |
| **symbols-bytes**          |     16,474,666 |     16,474,676 | 🡕 10            |   0.00% |
| **symbols-number**         |        376,426 |        376,427 | 🡕 1             |   0.00% |
| **values-bytes**           |  6,856,506,288 |  6,316,585,560 | 🡖 539,920,728   |  -7.87% |
| **values-number**          |    285,687,762 |    263,191,065 | 🡖 22,496,697    |  -7.87% |

The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-instantiate ./nixos/release-combined.nix -A tested`:

| stat                       | before         | after          | Δ               | Δ%     |
|:---------------------------|---------------:|---------------:|:----------------|-------:|
| **cpuTime**                |        256.071 |        237.531 | 🡖 18.54         | -7.24% |
| **envs-bytes**             |  7,111,004,192 |  7,041,478,520 | 🡖 69,525,672    | -0.98% |
| **envs-elements**          |    346,236,940 |    339,588,487 | 🡖 6,648,453     | -1.92% |
| **envs-number**            |    271,319,292 |    270,298,164 | 🡖 1,021,128     | -0.38% |
| **gc-heapSize**            |  8,995,291,136 | 10,110,009,344 | 🡕 1,114,718,208 | 12.39% |
| **gc-totalBytes**          | 37,172,737,408 | 36,878,391,888 | 🡖 294,345,520   | -0.79% |
| **list-bytes**             |  1,886,162,656 |  1,886,163,472 | 🡕 816           |  0.00% |
| **list-concats**           |      6,898,114 |      6,898,114 | 0               |        |
| **list-elements**          |    235,770,332 |    235,770,434 | 🡕 102           |  0.00% |
| **nrAvoided**              |    328,829,821 |    326,618,157 | 🡖 2,211,664     | -0.67% |
| **nrFunctionCalls**        |    240,850,845 |    239,998,495 | 🡖 852,350       | -0.35% |
| **nrLookups**              |    144,849,632 |    142,126,339 | 🡖 2,723,293     | -1.88% |
| **nrOpUpdateValuesCopied** |    251,032,504 |    251,032,504 | 0               |        |
| **nrOpUpdates**            |     17,903,110 |     17,903,110 | 0               |        |
| **nrPrimOpCalls**          |    140,674,913 |    139,485,975 | 🡖 1,188,938     | -0.85% |
| **nrThunks**               |    294,643,131 |    288,678,022 | 🡖 5,965,109     | -2.02% |
| **sets-bytes**             |  9,464,322,192 |  9,456,172,048 | 🡖 8,150,144     | -0.09% |
| **sets-elements**          |    377,474,889 |    377,134,877 | 🡖 340,012       | -0.09% |
| **sets-number**            |     50,615,607 |     50,616,875 | 🡕 1,268         |  0.00% |
| **sizes-Attr**             |             24 |             24 | 0               |        |
| **sizes-Bindings**         |              8 |              8 | 0               |        |
| **sizes-Env**              |             16 |             16 | 0               |        |
| **sizes-Value**            |             24 |             24 | 0               |        |
| **symbols-bytes**          |      3,147,102 |      3,147,064 | 🡖 38            | -0.00% |
| **symbols-number**         |         82,819 |         82,819 | 0               |        |
| **values-bytes**           | 11,147,448,768 | 10,996,111,512 | 🡖 151,337,256   | -1.36% |
| **values-number**          |    464,477,032 |    458,171,313 | 🡖 6,305,719     | -1.36% |
2019-04-11 19:21:38 -04:00
Frederik Rietdijk 2fcb11a244 Merge staging-next into master 2019-03-01 09:06:20 +01:00
Matthew Bauer b86e62d30d llvm: support cross compilation with useLLVM flag
You can build (partially) with LLVM toolchain using the useLLVM flag.
This works like so:

  nix-build -A hello --arg crossSystem '{ system =
    "aarch64-unknown-linux-musl"; useLLVM = true }'

also don’t separate debug info in lldClang

It doesn’t work currently with that setup hook. Missing build-id?
2019-02-26 19:45:35 -05:00
Daiderd Jordan 7ec53a932f
stdenv: only set __darwinAllowLocalNetworking on darwin
This is a darwin only nix attribute for sandbox builds, it can be
ignored on other platforms to avoid unnecessary rebuilds.
2019-02-14 21:38:08 +01:00
Matthew Bauer 5c09d977c7 Merge remote-tracking branch 'origin/master' into staging 2019-02-09 12:14:06 -05:00
Matthew Bauer c6f8f8d98d make-derivation: only modify name when name is given
This preserves Nix’s native error handling of missing name:

  error: derivation name missing
2019-02-04 09:15:03 -05:00
Vincent Laporte 0d17ecce2c
mkDerivation: cleaner handling of the name argument 2019-02-01 16:02:42 +00:00
Vladimír Čunát 8ba516664b
Merge branch 'staging-next' into staging 2019-02-01 09:42:53 +01:00
Vladimír Čunát 5effa4e0f9
Merge branch 'master' into staging-next
Comments on conflicts:
- llvm: d6f401e1 vs. 469ecc70 - docs for 6 and 7 say the default is
  to build all targets, so we should be fine
- some pypi hashes: they were equivalent, just base16 vs. base32
2019-02-01 09:22:29 +01:00
Matthew Bauer 8020bd6869
Merge pull request #35884 from dtzWill/fix/man-in-outputsToInstall
default to including "man" in outputsToInstall
2019-01-31 14:55:12 -05:00
Matthew Bauer e2fe4c2d49 make-derivation: fix ordering of conditionals
cross should have higher precedence
2019-01-28 11:38:30 -05:00
Matthew Bauer 9fd1c170cc make-derivation: try to fix stdenv.cc == null conditional
sometimes this gets an infinite recursion error
2019-01-28 10:41:19 -05:00
Matthew Bauer 7e589e5594 make-derivation: fix position in trace
For a long time now, tracing has been broken in Nixpkgs. So when you
have an eval error you would get something like this:

  error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:11:
  while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
  while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:

This is coming from how Nix handles string context and how
make-derivation messes with the "name" attribute. This commit should
restore the old behavior so you get a nice line number like:

  error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/applications/misc/hello/default.nix:4:3:
  while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
  while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:

NOTE: This will still be broken for cross compilation due to the
prefixes we are adding to name.
2019-01-28 09:25:47 -05:00
Matthew Bauer d54fbbebbb
Merge pull request #49552 from matthewbauer/setup-dedupe
setup.sh: avoid running the same hook twice
2019-01-27 18:22:41 -05:00
Matthew Bauer 9172c1eee2 setup.sh: avoid running the same hook twice
In strictDeps=false, we don’t want the same package hook to be run
twice, otherwise we get duplicates in some flags.

Fixes #41340
2019-01-27 17:49:13 -05:00
Matthew Bauer ae16dd1a15
stdenv/make-derivation: don't hide broken packages
This behavior ended up breaking the handleEvalIssue functionality by hiding those packages. So something like this:

$ nix-env -iA nixpkgs.zoom-us

would silently fail, without telling the user how to fix it! Regardless, this "bug" should be handled in Nix - not Nixpkgs.

Fixes #38952.
2019-01-27 17:02:17 -05:00
Matthew Bauer bd62ac6b88
Merge pull request #54652 from matthewbauer/shell-in-flags-array
setup.sh: put SHELL in flagsArray
2019-01-27 13:55:54 -05:00
Matthew Bauer 329913f733 setup.sh: put SHELL in flagsArray
We don’t want to modify makeFlags, that is given to us by our
environment. Adding to it could lead to duplicates after repeated use.

Fixes #27533
2019-01-26 21:14:36 -05:00
Matthew Bauer 8babcc3d44
Merge branch 'master' into fix/man-in-outputsToInstall 2019-01-26 10:51:12 -05:00
Frederik Rietdijk a1a5ea5943 stdenv: make checkInputs native
We can't run the checkPhase when build != host, so we may as well make
the checkInputs native.

This signicantly improves the situation of Python packages when enabling
strictDeps.
2019-01-13 14:43:18 +01:00
Orivej Desh (NixOS) 9a21967f0a
stdenv: prune libtool files by default (#51767)
See the motivation in fd97db43bc (#41819).
2019-01-11 13:20:46 +00:00
Daniel Goertzen 1c10efc912 add generic x86_32 support (#52634)
* add generic x86_32 support

- Add support for i386-i586.
- Add `isx86_32` predicate that can replace most uses of `isi686`.
- `isi686` is reinterpreted to mean "exactly i686 arch, and not say i585 or i386".
- This branch was used to build working i586 kernel running on i586 hardware.

* revert `isi[345]86`, remove dead code

- Remove changes to dead code in `doubles.nix` and `for-meta.nix`.
- Remove `isi[345]86` predicates since other cpu families don't have specific model predicates.

* remove i386-linux since linux not supported on that cpu
2019-01-06 12:57:36 -06:00
Jörg Thalheim 1b146a8c6f
treewide: remove paxutils from stdenv
More then one year ago we removed grsecurity kernels from nixpkgs:
https://github.com/NixOS/nixpkgs/pull/25277

This removes now also paxutils from stdenv.
2018-12-22 12:55:05 +01:00
Graham Christensen a375d4bfc3
stdenv: shorten evaluation errors when in Hydra
Hydra's page showing evaluation errors is about a mile long, showing
buckets of user-friendly errors, like this:

    in job ‘seyren.aarch64-linux’:
    Package ‘oraclejre-8u191’ in /nix/store/fa9zzkbljkvdavwzirkrr5irg25ymbjl-source/pkgs/development/compilers/oraclejdk/jdk-linux-base.nix:71 has an unfree license (‘unfree’), refusing to evaluate.

    a) For `nixos-rebuild` you can set
      { nixpkgs.config.allowUnfree = true; }
    in configuration.nix to override this.

    b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
      { allowUnfree = true; }
    to ~/.config/nixpkgs/config.nix.

    in job ‘jetbrains.webstorm.x86_64-linux’:
    Package ‘webstorm-2018.3.1’ in /nix/store/fa9zzkbljkvdavwzirkrr5irg25ymbjl-source/pkgs/applications/editors/jetbrains/default.nix:230 has an unfree license (‘unfree’), refusing to evaluate.

    a) For `nixos-rebuild` you can set
      { nixpkgs.config.allowUnfree = true; }
    in configuration.nix to override this.

    b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
      { allowUnfree = true; }
    to ~/.config/nixpkgs/config.nix.

This makes it extremely hard to find actual issues in the output. This
patch set makes the output much more condensed in Hydra:

    Failed to evaluate nifticlib-2.0.0: «unsupported»: is not supported on ‘x86_64-apple-darwin’
    Failed to evaluate dmd-2.081.2: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
    Failed to evaluate dmdBuild-2.081.2: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
    Failed to evaluate ldc-1.11.0: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
    Failed to evaluate ldcBuild-1.11.0: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
    Failed to evaluate ldc-0.17.5: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
    Failed to evaluate ldcBuild-0.17.5: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
2018-12-18 10:50:53 -05:00
Piotr Bogdan 44c9c27d54 stdenv: prune libtool files by default 2018-12-09 22:45:15 +00:00
Jan Tojnar c5881ec2c9
Merge remote-tracking branch 'upstream/master' into staging 2018-11-30 20:09:45 +01:00
Matthew Bauer 28e2277305 make-derivation: remove selfConsistent check
version is set in lots of places but might not need to be in a name.

Alternative to #50364.
2018-11-30 17:58:33 +01:00
Frederik Rietdijk 1828a5c5ba Merge master into staging-next 2018-11-30 17:46:21 +01:00
Matthew Bauer e6834171b9
Merge pull request #51183 from matthewbauer/make-derivation-no-assertions
Fix breaking changes to make-derivation
2018-11-29 19:35:55 -06:00
Vaibhav Sagar b5504e26b0 pkgs/stdenv/generic/make-derivation.nix: s/targetPlatform/hostPlatform/ 2018-11-29 17:54:59 -05:00
Matthew Bauer a56fe056ec make-derivation: don’t add host suffix if there is no c compiler
Some trivial builders use the name attr to choose the exec name
produced. For example nixos-install,

  nixos-install = makeProg {
    name = "nixos-install";
    src = ./nixos-install.sh;
    nix = config.nix.package.out;
    path = makeBinPath [ nixos-enter ];
  };

When cross compiling, this puts the prog in,

  /bin/nixos-install-powerpc64le-unknown-linux-gnu
2018-11-28 12:44:10 -06:00
Matthew Bauer d64f288154 make-derivation: don’t assert on separateDebugInfo
When separateDebugInfo = true & !stdenv.hostPlatform.isLinux, we
always gave an error. There is no reason to do this. Instead, just
don’t add separate debug info when we aren’t on Linux.
2018-11-28 12:29:00 -06:00
Matthew Bauer 13e45aeed3 make-derivation: remove selfConsistent check
version is set in lots of places but might not need to be in a name.

Alternative to #50364.
2018-11-28 12:26:21 -06:00
Vladimír Čunát a5de78b7d7
Merge branch 'master' into staging-next 2018-11-26 10:28:00 +01:00
Matthew Bauer 7eeb02d47b Revert "make-derivation: add disallowedReferences in strictDeps"
This reverts commit 8dbfb61e46.
Also reverts commit fc99c337ed.

Fixes #50915
2018-11-25 15:53:28 -06:00
Frederik Rietdijk c31cb577ae Merge master into staging-next 2018-11-22 09:57:08 +01:00
Matthew Bauer fc99c337ed make-derivation: don’t disallow propagated native inputs
propagateNativeBuildInputs will end up going in the output derivation.
This case is allowed to end up in references because of that. Sorry
for the disruption!

Fixes #50865
2018-11-21 09:39:47 -06:00
Daiderd Jordan c9223a17bc
Revert "patch-shebangs: use --build for auto patch shebangs"
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.

This reverts commit eb7c50a993.
2018-11-14 23:37:31 +01:00
Matthew Bauer f19bb8321a
Merge pull request #35304 from volth/patch-97
[staging] substitute() print warning if does nothing
2018-11-12 12:52:51 -06:00
Frederik Rietdijk 3b052406ea Merge staging-next into staging 2018-11-12 19:01:36 +01:00
Frederik Rietdijk daf3297cb4 Merge master into staging-next 2018-11-12 18:59:08 +01:00
Léo Gaspard 2986ce16a8
meta.tests: rename into passthru.tests
Nix currently rejects derivations in `meta` values. This works around
that limitation by using `passthru` instead.

Closes https://github.com/NixOS/nixpkgs/issues/50230
2018-11-11 23:11:46 +09:00
Léo Gaspard cae5598611
meta.tests: drop meta.needsVMSupport
Its job is already handled by `requiredSystemFeatures`
2018-11-11 23:11:46 +09:00
Léo Gaspard 83b27f60ce
tests: split into a separate all-tests.nix file
This will make the list much easier to re-use, eg. for `nixosTests`

The drawback is that this approaches makes the
```
nix-build release.nix -A tests.opensmtpd.x86_64-linux
```
command about twice as slow (3s to 6s): it now has to evaluate `nixpkgs`
once for each architecture, instead of just having the hardcoded list of
tests that allowed to say “ok just evaluate for x86_64-linux”.

On the other hand, complete evaluation of `release.nix` should be much
faster because we no longer import `nixpkgs` for each test: testing with
the following command went from 30s to 18s, and that's just for a few
tests.
```
time nix-instantiate --eval --strict nixos/release.nix -A tests.nat
```
I initially wanted to test on the whole `release.nix`, but there are too
many broken tests and it takes too long to eval them all, especially
compared to the fact that the current implementation breaks some setup.

Given developers can just `nix-build nixos/tests/my-test.nix`, it sounds
like an overall win.
2018-11-11 23:11:46 +09:00
Frederik Rietdijk 1d3bff25db Merge staging-next into staging 2018-11-11 14:28:08 +01:00
Frederik Rietdijk 53d00c3351 Merge master into staging-next 2018-11-10 11:08:54 +01:00
Matthew Bauer c8aff96110
Merge pull request #49608 from matthewbauer/cross-patch-shebangs-2
Restore cross-patch-shebangs branch
2018-11-07 13:37:02 -06:00
Matthew Bauer 6d531f3541 make-derivation: enable pie hardening with musl
Fixes #49071

On ld.gold, we produce broken executables when linking with the Musl
libc. This appears to be a known bug when using ld.gold and Musl. This
thread describes the workaround as enabling PIE when using ld.gold and
Musl:

https://www.openwall.com/lists/musl/2015/05/01/5

By default we don’t enable PIE to avoid breaking things. But in the
Musl case we are breaking things by not enabling PIE. So this adds a
special case for defaultHardeningFlags which keeps the pie hardening
for everything. Any packages that break with PIE can add the pie flag
to disableHardeningFlags array (a no-op for now on anything but Musl).
2018-11-07 15:24:51 +01:00
Patrick Hilhorst 320c9c10de
make-derivation: use pname-version as default name if both are present 2018-11-06 00:04:21 +01:00