Commit graph

69 commits

Author SHA1 Message Date
deliciouslytyped a71e906e3a trivial-builders: refactor writeTextFile to be overridable
This fixes #126344, specifically with the goal of enabling overriding the
checkPhase argument. See `design notes` at the end for details.

This allows among other things, enabling bash extension for the `checkPhase`.
Previously using such bash extensions was prohibited by the `writeShellScript`
code because there was no way to enable the extension in the checker.

As an example:

```nix
(writeShellScript "foo" ''
  shopt -s extglob
  echo @(foo|bar)
'').overrideAttrs (old: {
  checkPhase = ''
    # use subshell to preserve outer environment
    (
      export BASHOPTS
      shopt -s extglob
      ${old.checkPhase}
    )
  '';
})
```

This commit also adds tests for this feature to `pkgs/tests/default.nix`,
under `trivial-overriding`. The test code is located at
`pkgs/build-support/trivial-builders/test-overriding.nix`.

Design notes:
-------------

Per discussion with @sternenseemann, the original approach of just wrapping
`writeTextFile` in `makeOverridable` had the issue that combined with `callPackage`
in the following form, would shadow the `.override` attribute of the `writeTextFile`:

```nix
with import <nixpkgs>;
callPackage ({writeShellScript}: writeShellScript "foo" "echo foo")
```

A better approach can be seen in this commit, where `checkPhase` is moved
from an argument of `writeTextFile`, which is substituted into `buildCommand`,
into an `mkDerivation` argument, which is substituted from the environment
and `eval`-ed. (see the source)

This way we can simple use `.overideAttrs` as usual, and this also makes
`checkPhase` a bit more conformant to `mkDerivation` naming, with respect to
phases generally being overridable attrs.

Co-authored-by: sterni <sternenseemann@systemli.org>
Co-authored-by: Naïm Favier <n@monade.li>
2021-06-18 01:39:59 +02:00
Robert Hensing 047fe2fe23 emptyFile, emptyDirectory: init 2021-06-12 17:28:00 +02:00
Robert Hensing cc60f81e69 writeDirectReferencesToFile: init 2021-05-15 17:04:25 +02:00
Arnout Engelen b68130fd2c
test-utilities: version test
Extract 'version test' to a reusable test utility as discussed in
https://github.com/NixOS/nixpkgs/pull/119636#issuecomment-826137021 and
2021-05-07 09:53:35 +02:00
sternenseemann de78745bdd trivial-builders: rework runCommand' into runCommandWith && expose
runCommandWith receives an attribute set with options which previously
were positional arguments of runCommand' and a buildCommand. This
allows for overriding the used stdenv freely (so stuff like
llvmPackages.stdenv can be used). Additionally the possibility to change
arguments passed to stdenv.mkDerivation is made more explicit via the
derivationArgs argument.
2021-03-18 19:56:40 +01:00
sternenseemann b398d00903 trivial-builders: merge passAsFile with env in runCommand'
Previously it was awkward to use the runCommand-variants with
passAsFile as a double definition of passAsFile would potentially
break runCommand: passAsFile would overwrite the previous definition,
defeating the purpose of setting it in runCommand in the first place.
This is now fixed by concatenating the [ "buildCommand" ] list with
one the one from env, if present.

Adjust buildEnv where passAsFile = null; was passed in some cases,
breaking evaluation since it'd evaluate to [ "buildCommand" ] ++ null.
2021-03-18 19:56:40 +01:00
Vladimír Čunát 3937923f81
nixos/systemd.tmpfiles.packages: fix an edge case
symlinkJoin can break (silently) when the passed paths contain symlinks
to directories.  This should work now.

Down-side: when lib/tmpfiles.d doesn't exist for some passed package,
the error message is a little less explicit, because we never get
to the postBuild phase (and symlinkJoin doesn't provide a better way):
/nix/store/HASH-NAME/lib/tmpfiles.d: No such file or directory

Also, it seemed pointless to create symlinks for whole package trees
and using only a part of the result (usually very small part).
2020-08-16 10:23:53 +02:00
Silvan Mosberger c60e559e13
trivial-builders: Sanitize derivation name
This then supports using functions like writeShellScriptBin with script names
that would be invalid as derivation names
2020-04-15 00:31:57 +02:00
(cdep)illabout 55e1529335 symlinkJoin: add more documentation 2020-02-18 16:35:28 +01:00
(cdep)illabout ae607de23d linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a list of derivations 2020-02-18 16:35:28 +01:00
Bas van Dijk 4e0a6bc4be symlinkJoin: fix "bash: Argument list too long" error
When the `paths` argument is too big `symlinkJoin` will fail with:

```
while setting up the build environment: executing '/nix/store/rm1hz1lybxangc8sdl7xvzs5dcvigvf7-bash-4.4-p23/bin/bash': Argument list too long
```

This is fixed by passing `paths` as a file instead of as an
environment variable.
2020-02-17 11:45:44 +01:00
Profpatsch c5c5465fe4 pkgs/build-support/trivial-builders: remove runCommandCCLocal
We shouldn’t force the user to have a C compiler in scope, just
because the derivation is forced to build locally. That can’t be
counted as “lightweight” anymore.

Co-Authored-By: Silvan Mosberger<contact@infinisil.com>
2019-12-04 21:17:01 +01:00
Profpatsch 8deaf41d60 pkgs/build-support/trivial-builders: add runCommandLocal
A definition I’ve been copy-pasting everywhere so far, so it’s finally
time to add it to nixpkgs.

I’m using a remote builder for my regular nix builds, so trivial
`runCommand`s which first try a substitution and then copy the inputs
to the builder to run for 0.2s are quite noticable.

If we just always build these, we gain some build time, so let’s make
it easy to switch from remote to local.
2019-12-04 21:17:01 +01:00
Bas van Dijk 05393ca217 trivial-builders: add the applyPatches function
applyPatches applies a list of patches to a source directory.

For example to patch nixpkgs you can use:

  applyPatches {
    src = pkgs.path;
    patches = [
      (pkgs.fetchpatch {
        url = "1f770d2055.patch";
        sha256 = "1nlzx171y3r3jbk0qhvnl711kmdk57jlq4na8f8bs8wz2pbffymr";
      })
    ];
  }
2019-09-02 11:18:52 +02:00
Robert Helgesson 6900a327ad
trivial-builders: support '/' in writeTextDir
Before one would get the following error

    nix-repl> pkgs.writeTextDir "share/my-file" "foo"
    error: invalid character '/' in name 'share/my-file'

Fixes #50347
2019-07-13 17:03:41 +02:00
Erik Nygren 4e5b10d98c trivial-builders: Fix writeShellScript example (#63888) 2019-06-29 14:50:02 +00:00
Jaakko Luttinen fa95878fd3 writeScript: fix docstring
Fixes #59299

See:
b23dbb1a5d
2019-06-09 12:45:31 +02:00
deliciouslytyped 103ab24e94 trivial-builders: add writeShellScript and minor cleaning
Add writeShellScript
Small whitespace additions
Fix "Example:" docstring sections for some of the writeScript functions to use the correct function
2019-05-12 19:40:01 +02:00
Niklas Hambüchen a0fa0cd8cf trivial-builders: Fix runCommand examples 2019-04-19 06:29:42 +02:00
Niklas Hambüchen c4c2aa6586 trivial-builders: Fix outdated comment on writeTextFile 2019-04-08 01:12:50 +02:00
Michael Eden 53b0027fb4 writeShellScriptBin: use build-time shell in check phase 2019-04-04 13:31:51 -04:00
Jörg Thalheim dadc7eb329
treewide: use runtimeShell instead of stdenv.shell whenever possible
Whenever we create scripts that are installed to $out, we must use runtimeShell
in order to get the shell that can be executed on the machine we create the
package for. This is relevant for cross-compiling. The only use case for
stdenv.shell are scripts that are executed as part of the build system.
Usages in checkPhase are borderline however to decrease the likelyhood
of people copying the wrong examples, I decided to use runtimeShell as well.
2019-02-26 14:10:49 +00:00
volth f07c113ddc
linkFarm: allowSubstitutes = false
trivial builder
2019-02-08 00:21:24 +00:00
volth 97870258b2
linkFarm: hacky quoting -> escapeShellArg 2019-01-31 17:26:49 +00:00
Samuel Leathers 024eb9a5a5 trivial builders: adding usage documentation for functions 2018-10-02 22:09:09 +02:00
Alyssa Ross 4af7278bc9 lib: ensure directories of linkFarm links exist (#45628)
There's no reason `linkFarm` can't be used for symlinks in
subdirectories, except that currently it doesn't ensure the directory
of the link exists. This backwards-compatible change expands the utility
of the function.
2018-09-01 14:53:23 +02:00
Matthew Bauer f194659ddb trivial-builders: disallow sub in requireFile
The requireFile call was being substituted from the binary cache. We
do not want this to happen as the user needs to download the file
themselves.
2018-06-26 21:49:08 -04:00
Uri Baghin 274bb96073 bazel: add darwin support 2018-06-12 23:23:51 -04:00
John Ericson 9ec53a397f requireFile: Use stdenvNoCC
There's no need for a C compiler
2018-05-21 19:26:36 -04:00
Alexandre Esteves d273691f6d trivial-builders.nix: support directories in requireFile
Allow recursive hash for requireFile so it can support directories (#40568)
2018-05-16 01:41:13 -05:00
Linus Heckemann 4a30f2efec requireFile: exit with non-zero error message
Since the script running is a failure condition, we should fail the
build properly, not leaving it up to the missing output to determine
that the build went wrong.  This should partly address #38952 — nix
build will print out the build log on non-zero exits.
2018-04-15 13:58:05 +01:00
Orivej Desh 170a964815 makeSetupHook: make the default name "hook" overridable
for occasional convenience while looking at drv paths,
such as in the output of nix-build and nix-diff.
2017-12-19 01:41:31 +00:00
John Ericson da19c34d0f stdenv setup: Always use both propagated files
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.

This is an incremental step to fully keeping the dependencies separate
in all cases.
2017-11-21 10:44:44 -05:00
Nikita Uvarov cb593780f3 trivial-builders.nix: fix writeShellScriptBin
The check should be performed on the target shell script, not on the output directory.
2017-10-06 10:55:36 +02:00
Sergey Mironov f49c2fbf7a
trivial-builders.nix: add writeShellScriptBin builder 2017-08-01 20:23:08 -04:00
John Ericson f6f40e3fe5 stdenv-setup and misc pkgs: Revert to space-deliminated propagated-* files
We cannot switch to line-delimited yet, because certain Nix commands do
not read in the entire file, but just the first line.
2017-07-26 09:07:55 -04:00
John Ericson 3cb745d5a6 stdenv: Store one package per line in nix-support/propagated-*
This makes those files a bit easier to read. Also, for what it's worth,
it brings us one baby step closer to handling spaces in store paths.

Also, I optimized handling of many transitive deps with read. Probably,
not very beneficial, but nice to enforce the pkg-per-line structure.
Doing so let me find much dubious code and fix it.

Two misc notes:

 - `propagated-user-env-packages` also needed to be adjusted as
   sometimes it is copied to/from the propagated input files.

 - `local fd` should ensure that file descriptors aren't clobbered
   during recursion.
2017-07-10 13:32:13 -04:00
Ryan Trinkle 0916c26ed6 symlinkJoin: eliminate lndir spam
By default lndir outputs the name of each subdirectory as it descends into it; this commit gives it the -silent flag to disable this behavior
2017-05-27 16:19:34 -04:00
Michael Raskin c94e44b0ae linkFarm: prefer local build 2016-12-09 11:26:53 +01:00
Eelco Dolstra 97bfc2fac9 runCommand: Use stdenvNoCC
This ensures that most "trivial" derivations used to build NixOS
configurations no longer depend on GCC. For commands that do invoke
gcc, there is runCommandCC.
2016-09-29 13:06:43 +02:00
Eelco Dolstra c5ddb7dd56 Move useSetUID to pam_usb, the only place where it's used 2016-09-29 13:05:28 +02:00
Eelco Dolstra 030e20f759 Add a warning about using requireFile 2016-09-29 13:05:28 +02:00
Vladimír Čunát ab9515092b symlinkJoin: truly pass the defaulted parameters
The catch is that in nix the @-pattern binds the set that was *explicitly*
passed to the function, i.e. default values are *not* taken into account.
2016-07-14 15:36:53 +02:00
Domen Kožar 7a5b85cdda pkgs.runCommand: passAsFile (buildCommand can be very long)
Close #15803. This avoids the error:

while setting up the build environment: executing
‘/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin/bash’:
Argument list too long

Note: I wanted to make it optional based on buildCommand length,
but that seems pointless as I'm sure it's less performant.

Amended by vcunat:
https://github.com/NixOS/nixpkgs/pull/15803#issuecomment-224841225
2016-06-10 10:49:26 +02:00
Nikolay Amiantov ebe1cbe0da symlinkJoin: allow arbitrary additional attributes 2016-05-27 13:42:22 +03:00
Nikolay Amiantov dfe608c8a2 symlinkJoin: accept set as an argument with additional options 2016-04-26 15:37:42 +03:00
Vladimír Čunát ab0bc1ecaf symlinkJoin: preferLocalBuild && !allowSubstitutes 2016-03-11 15:59:18 +01:00
Profpatsch 82fa1a796b lib/copyPathToStore: annotate docstring 2016-03-01 15:26:35 +01:00
Vladimír Čunát 4b581903b3 requireFile: preferLocalBuild = true
There's no point trying to "distribute" showing the error message.
2016-02-19 13:47:50 +01:00
Domen Kožar 36057827f6 requireFile: remove colon from the url for easier copy/paste 2016-01-16 21:58:40 +01:00