Commit graph

229 commits

Author SHA1 Message Date
Eelco Dolstra 6b3cef2246 Remove tracePhases
This has not been used in a long time.
2017-09-07 22:15:37 +02:00
Orivej Desh f4044c1ccc stdenv-setup: list environment variables with awk 2017-09-03 12:57:08 +00:00
Orivej Desh a09d9e7cd4 stdenv-setup: fix substituteAll with set -eu
Environment variable filter in substituteAll was not precise and produced
undefined and invalid variable names.  Vladimír Čunát tried to fix that in [1],
but `env -0` did not work during Darwin bootstrap, so [2] reverted this change
and replaced an error due to invalid variables with a warning.  Recently in #28057
John Ericson added `set -u` to `setup.sh` and undefined variables made the setup
fail during e.g. `nix-build -A gnat` with `setup: line 519: !varName: unbound
variable`.

[1] 62fc8859c1
[2] 81df035429
2017-09-03 12:57:08 +00:00
John Ericson 81194eef45 stdenv-setup: Use set -u as much as possible
Older bash version, like those in the bootstrap tools and on macOS,
currently confuse variables defined as an empty array with undefined
variables. `${foo+"${foo[@]}"}` will prevent `set -u` problems with
empty arrays and older without making a single '' in the empty case.

Care is taken to `set +u` when running hooks so as to not break existing
packages.
2017-08-23 15:57:56 -04:00
Tuomas Tynkkynen 7320fa9d45 Revert "stdenvs: Distinguish between extraBuildInputs and extraNativeBuildInputs"
This reverts commit eeabf85780.

This change suddenly makes tons of stdenv internals visible in
nativeBuildInputs of every derivation, which doesn't seem desirable.
E.g:

````
nix-repl> hello.nativeBuildInputs
[ «derivation /nix/store/bcfkyf6bhssxd2vzwgzmsbn7b5b9rpxc-patchelf-0.9.drv»
  «derivation /nix/store/4wnshnz9wwanpfzcrdd76rri7pyqn9sk-paxctl-0.9.drv»
  << snip 10+ lines >>
  «derivation /nix/store/d35pgh1lcg5nm0x28d899pxj30b8c9b2-gcc-wrapper-6.4.0.drv»
]
````
2017-08-18 13:21:56 +03:00
John Ericson eeabf85780 stdenvs: Distinguish between extraBuildInputs and extraNativeBuildInputs
Additionally, instead of pulling them from `setup.sh`, route them via
Nix. This gets us one step closer to making stdenv be a plain attribute
set instead of a derivation.
2017-08-15 18:24:54 -04:00
Linus Heckemann 17753fa005 stdenv: fix typo in setup.sh 2017-08-09 17:33:02 +01:00
John Ericson 42f35503b5 cc-wrapper: Make hygienic
See the added comments for what exactly has been done.
2017-08-07 03:05:50 -04:00
John Ericson 820e4021d3 stdenv-setup: Remove any declare -g
This is invalid before bash-4.2, affecting bash used impurely in
nix-shell on MacOS.
2017-07-26 09:11:18 -04:00
John Ericson ea7d13cf1a stdenv-setup and misc hooks: Work with bash-3.4 for MacOS nix-shell
This is a temporary measure until this impurity is removed from Nix.
2017-07-26 09:08:01 -04:00
John Ericson f6f40e3fe5 stdenv-setup and misc pkgs: Revert to space-deliminated propagated-* files
We cannot switch to line-delimited yet, because certain Nix commands do
not read in the entire file, but just the first line.
2017-07-26 09:07:55 -04:00
John Ericson 34c0ba498c stdenv-setup: Add quotes that don't do anything for consistency.
@vcunat and others rightly point out that it's easier to quote always,
than learn Bash's idiosyncrasies enough to know when it doesn't make a
difference.

This reverts commit 2743078f66, which
removes quotes that don't do anything, and then goes further adding
even more quotes.
2017-07-25 14:36:00 -04:00
Eelco Dolstra 6669a3b477
stdenv: Remove log nesting
Nix/Hydra no longer support pretty printing of logs, so this is no
longer useful.
2017-07-24 14:30:46 +02:00
John Ericson 30a1420414 stdenv-setup: Pull out and explain 3-part printing of commands
@Dezgeg made the good point that the reasons for doing this were not at
all intuitive.
2017-07-13 16:31:39 -04:00
John Ericson 2743078f66 stdenv-setup: Remove useless quotes
foo=$1 surprisingly doesn't need quotes in Bash. Word splits are only
syntactic in string variable (not array var!) assignments.
2017-07-13 14:59:53 -04:00
John Ericson 273a4c1c78 stdenv-setup: Combine [[ .. ]] && [[ .. ]] into one [[ .. && .. ]]
Also remove useless quotes on same line
2017-07-13 14:57:16 -04:00
John Ericson 5d693c84d2 stdenv-setup: Clean up 'substitute()' for style and error handling
It now blows up on null byte in file (rather than silently truncating),
and invalid arguments (rather than silently skipping).
2017-07-12 17:47:20 -04:00
John Ericson 5d4efb2c81 stdenv-setup: Misc improvements as directed by ShellCheck
I took some liberties with the flags-echoing code to make it more
concise and correct. Also, a few warnings in findInputs and friends I
skipped because I am going to rewrite those anyways.

Thanks @grahamc for telling me about this great linter!
2017-07-12 15:31:10 -04:00
John Ericson 8d76effc17 stdenv-setup: Make the package accumulators associative arrays instead of strings
This is generally cleaner: less eval, less worrying about separators,
and probably also faster. I got the idea from that python wrapper
script.
2017-07-12 15:30:56 -04:00
John Ericson 3cb745d5a6 stdenv: Store one package per line in nix-support/propagated-*
This makes those files a bit easier to read. Also, for what it's worth,
it brings us one baby step closer to handling spaces in store paths.

Also, I optimized handling of many transitive deps with read. Probably,
not very beneficial, but nice to enforce the pkg-per-line structure.
Doing so let me find much dubious code and fix it.

Two misc notes:

 - `propagated-user-env-packages` also needed to be adjusted as
   sometimes it is copied to/from the propagated input files.

 - `local fd` should ensure that file descriptors aren't clobbered
   during recursion.
2017-07-10 13:32:13 -04:00
John Ericson e57a220f81 stdenv, swift: Use local fd in is* bash functions for hygiene 2017-07-07 11:40:07 -04:00
John Ericson a14cf06182 stdenv: Harden hook runners
Instead of eval, use a "nameref" to get the name of the array and
iterate with that. Also, make the for-loop parameter a local variable,
too.
2017-07-07 11:35:09 -04:00
Eelco Dolstra 94d164dd7f
Add a setup hook for detecting $TMPDIR references in RPATHs and wrapper scripts 2017-05-04 20:23:57 +02:00
Tuomas Tynkkynen ce56c99edc mkDerivation: Don't pass buildInputs to stdenv builder in nativeBuildInputs
When not cross compiling, nativeBuildInputs and buildInputs have
identical behaviour. Currently that is implemented by having
mkDerivation do a concatenation of those variables in Nix code and pass
that to the builder via the nativeBuildInputs attribute.

However, that has some annoying side effects, like `foo.buildInputs`
evaluating to `[ ]` even if buildInputs were specified in the nix
expression for foo.

Instead, pass buildInputs and nativeBuildInputs in separate variables as
usual, and move the logic of cross compilation vs. native compilation to
the stdenv builder script. This is probably a tiny bit uglier but
fixes the previous problem.

Issue #4855.
2017-03-02 03:26:48 +02:00
Vladimír Čunát af38c05587 stdenv stripHash(): fixup after #19324 2016-10-12 23:45:30 +02:00
Profpatsch bef6bef0d2
stdenv/stripHash: print to stdout, not to variable
`stripHash` documentation states that it prints out the stripped name to
the stdout, but the function stored the value in `strippedName`
instead.

Basically all usages did something like
`$(stripHash $foo | echo $strippedName)` which is just braindamaged.
Fixed the implementation and all invocations.
2016-10-11 18:34:36 +02:00
Nikolay Amiantov adaee7352b stdenv: leave SSL_CERT_FILE in shells (#15571) 2016-09-01 20:50:08 +02:00
Eelco Dolstra 8a84fc0217 Tweak error message 2016-08-26 18:58:49 +02:00
Robin Gloster b6c204f088
stdenv substitute: fail on non-existant input file
fixes #9744
2016-08-26 16:27:36 +00:00
Domen Kožar 7a5b85cdda pkgs.runCommand: passAsFile (buildCommand can be very long)
Close #15803. This avoids the error:

while setting up the build environment: executing
‘/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin/bash’:
Argument list too long

Note: I wanted to make it optional based on buildCommand length,
but that seems pointless as I'm sure it's less performant.

Amended by vcunat:
https://github.com/NixOS/nixpkgs/pull/15803#issuecomment-224841225
2016-06-10 10:49:26 +02:00
Joachim Fasting 6648b04381
stdenv: fix paxmark
On Linux, paxctl's setup hook should overwrite the paxmark stub, but the
stub is defined after the setup hooks are sourced, so the stub ends up
overwriting the real function.  The result is that paxmark fails to do
anything.  The fix is to define the stub before any setup hooks are
sourced.  Thanks to @vcunat for figuring this out.

Closes #15492
2016-05-27 18:57:59 +02:00
Vladimír Čunát 81df035429 stdenv setup.sh: revert most of changes around #14907
I'm giving this up. Feel free to find some reasonable variant that works
at least on Linux and Darwin. Problems encountered:
- During bootstrap of Darwin stdenv `env -0` and some bash features
  don't work.
- Without `env -0` the contents of some multi-line phases is taken as
  variable declarations, which wouldn't typically matter, but the PR
  wanted to refuse bash-invalid names which would be occasionally
  triggered. This commit dowgrades that to a warning with explanation.
2016-05-12 04:53:37 +02:00
Vladimír Čunát 62fc8859c1 stdenv substituteAll: use yet another implementation
It turned out that process substitution fed into a while-cycle
isn't recognized during darwin bootstrap:
http://hydra.nixos.org/build/35382446/nixlog/1/raw

Also fix broken NIX_DEBUG output, noticed by abbradar.
2016-05-08 19:41:50 +02:00
Vladimír Čunát 9e0d0423fe stdenv substituteAll: use more robust code
The set/env fix in #14907 wasn't very good, so let's use a null-delimited
approach. Suggested by Aszlig.
In particular, this should fix a mass-breakage on Darwin, though I was
unable to test that.
2016-05-07 11:23:30 +02:00
Profpatsch a2d38bc7fc doc/stdenv.xml document substitution env variables
The filtering of environment variables that start with an uppercase
letter is documented in the manual.
2016-04-23 21:41:35 +02:00
Profpatsch 77fa336849 setup.hs: substitute uses only valid bash names
bash variable names may only contain alphanumeric ASCII-symbols and _,
and must not start with a number. Nix expression attribute names however
might contain nearly every character (in particular spaces and dashes).

Previously, a substitution that was not a valid bash name would be
expanded to an empty string. This commit introduce a check that throws
a (hopefully) helpful error when a wrong name is used in a substitution.
2016-04-23 17:54:32 +02:00
Vladimír Čunát d1df28f8e5 Merge 'staging' into closure-size
This is mainly to get the update of bootstrap tools.
Otherwise there were mysterious segfaults:
https://github.com/NixOS/nixpkgs/pull/7701#issuecomment-203389817
2016-04-07 14:40:51 +02:00
Charles Strahan bde82098b8 stdenv: don't complain about configure script not existing
Close #14335.
Since 89036ef76a, when a package doesn't include a configure script,
the build complains with:

    grep: : No such file or directory
    grep: : No such file or directory

This prevents that.
2016-04-02 20:52:19 +02:00
Vladimír Čunát ab15a62c68 Merge branch 'master' into closure-size
Beware that stdenv doesn't build. It seems something more will be needed
than just resolution of merge conflicts.
2016-04-01 10:06:01 +02:00
Vladimír Čunát 09af15654f Merge master into closure-size
The kde-5 stuff still didn't merge well.
I hand-fixed what I saw, but there may be more problems.
2016-03-08 09:58:19 +01:00
Eelco Dolstra 2040a9ac57 stdenv-linux: Ensure binutils comes before bootstrapTools in $PATH
Otherwise, when building glibc and other packages, the "strip" from
bootstrapTools is used, which doesn't recognise some tags produced by
the newer "ld" from binutils.
2016-02-28 01:13:15 +01:00
Nikolay Amiantov 39609a0c94 stdenv: set SSL_CERT_FILE only if it isn't already 2016-02-25 13:53:29 +03:00
Eelco Dolstra d71a4851e8 Don't try to apply patchelf to non-ELF binaries 2016-02-18 22:54:11 +01:00
Vladimír Čunát 89036ef76a stdenv: accept wider range of $configureScript options
Fixes #12632.

I think it's better to quote this variable in general, because it is
common and even documented to pass space-separated commands in there.
The greps should just fail in that case and `if` won't proceed
which seems fine for such cases, and it's certainly better than
passing additional unintended parameters to grep
(which was happening all the time before).
2016-02-03 17:15:11 +01:00
Vladimír Čunát ae74c356d9 Merge recent 'staging' into closure-size
Let's get rid of those merge conflicts.
2016-02-03 16:57:19 +01:00
Eelco Dolstra 917ca8920d Move setting $SSL_CERT_FILE to stdenv
Doing it in an openssl setup hook only works if packages have openssl
as a build input - it doesn't work if they're using a program linked
against openssl.
2016-02-03 13:59:10 +01:00
Eelco Dolstra f4553d7219 defaultUnpack: Preserve timestamps when copying files
Commit 6d928ab684 changed this to not
preserve timestamps. However, that results in non-determinism; in
particular, it gives us a broken $SOURCE_DATE_EPOCH (especially for
everything using fetchFromGitHub). Builds affected by timestamps <
1980 should be fixed in some other way (e.g. changing the timestamp to
some fixed date > 1980).
2016-01-26 16:24:32 +01:00
Vladimír Čunát 716aac2519 Merge branch 'staging' into closure-size 2016-01-19 09:55:31 +01:00
Eelco Dolstra f31fbadac3 Set a fallback default value for SOURCE_DATE_EPOCH
This is used by some build tools to provide reproducible builds. See

  https://reproducible-builds.org/specs/source-date-epoch/

for more info.

Later, we'll want to set this to a more intelligent value (such as the
most recent mtime of any source file).
2016-01-05 17:21:48 +01:00
Vladimír Čunát 1ebff73b88 stdenv/setup.sh: don't skip post-hooks (close #12032)
So far if no configure script is found or no makefile,
the rest of the phase is skipped, *including* post-hooks.
I find that behavior unexpected/unintuitive.

Earlier version of this patch had problems due to me assuming
that $configureScript is always a simple path, but that turned out
to be false in many cases, e.g. perl.
2016-01-05 09:34:02 +01:00