From 53fc7b8272cf762c57dd3e113863aa99feb35e5d Mon Sep 17 00:00:00 2001 From: Profpatsch Date: Thu, 17 Nov 2016 22:29:32 +0100 Subject: [PATCH] lib/generators: add manual documentation Restructures the functions reference a bit. --- doc/functions.xml | 801 ++++++++++++++++++++++++--------------------- lib/generators.nix | 7 +- 2 files changed, 428 insertions(+), 380 deletions(-) diff --git a/doc/functions.xml b/doc/functions.xml index 3850e58c016..70326936a57 100644 --- a/doc/functions.xml +++ b/doc/functions.xml @@ -8,252 +8,295 @@ The nixpkgs repository has several utility functions to manipulate Nix expressions. -
- pkgs.overridePackages +
+ Overriding - This function inside the nixpkgs expression (pkgs) - can be used to override the set of packages itself. - - - Warning: this function is expensive and must not be used from within - the nixpkgs repository. - - - Example usage: - - let - pkgs = import <nixpkgs> {}; - newpkgs = pkgs.overridePackages (self: super: { - foo = super.foo.override { ... }; - }; -in ... + Sometimes one wants to override parts of + nixpkgs, e.g. derivation attributes, the results of + derivations or even the whole package set. - - The resulting newpkgs will have the new foo - expression, and all other expressions depending on foo will also - use the new foo expression. - +
+ pkgs.overridePackages - - The behavior of this function is similar to config.packageOverrides. - - - - The self parameter refers to the final package set with the - applied overrides. Using this parameter may lead to infinite recursion if not - used consciously. - - - - The super parameter refers to the old package set. - It's equivalent to pkgs in the above example. - - - - Note that in previous versions of nixpkgs, this method replaced any changes from config.packageOverrides, - along with that from previous calls if this function was called repeatedly. - Now those previous changes will be preserved so this function can be "chained" meaningfully. - To recover the old behavior, make sure config.packageOverrides is unset, - and call this only once off a "freshly" imported nixpkgs: - - let - pkgs = import <nixpkgs> { config: {}; }; - newpkgs = pkgs.overridePackages ...; -in ... - - -
- -
- <pkg>.override - - - The function override is usually available for all the - derivations in the nixpkgs expression (pkgs). - - - It is used to override the arguments passed to a function. - - - Example usages: - - pkgs.foo.override { arg1 = val1; arg2 = val2; ... } - pkgs.overridePackages (self: super: { - foo = super.foo.override { barSupport = true ; }; -}) - mypkg = pkgs.callPackage ./mypkg.nix { - mydep = pkgs.mydep.override { ... }; -}) - - - - In the first example, pkgs.foo is the result of a function call - with some default arguments, usually a derivation. - Using pkgs.foo.override will call the same function with - the given new arguments. - - -
- -
- <pkg>.overrideAttrs - - - The function overrideAttrs allows overriding the - attribute set passed to a stdenv.mkDerivation call, - producing a new derivation based on the original one. - This function is available on all derivations produced by the - stdenv.mkDerivation function, which is most packages - in the nixpkgs expression pkgs. - - - - Example usage: - - helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec { - separateDebugInfo = true; -}); - - - - In the above example, the separateDebugInfo attribute is - overriden to be true, thus building debug info for - helloWithDebug, while all other attributes will be - retained from the original hello package. - - - - The argument oldAttrs is conventionally used to refer to - the attr set originally passed to stdenv.mkDerivation. - - - - Note that separateDebugInfo is processed only by the - stdenv.mkDerivation function, not the generated, raw - Nix derivation. Thus, using overrideDerivation will - not work in this case, as it overrides only the attributes of the final - derivation. It is for this reason that overrideAttrs - should be preferred in (almost) all cases to - overrideDerivation, i.e. to allow using - sdenv.mkDerivation to process input arguments, as well - as the fact that it is easier to use (you can use the same attribute - names you see in your Nix code, instead of the ones generated (e.g. - buildInputs vs nativeBuildInputs, - and involves less typing. + This function inside the nixpkgs expression (pkgs) + can be used to override the set of packages itself. - - -
- - -
- <pkg>.overrideDerivation - - - You should prefer overrideAttrs in almost all - cases, see its documentation for the reasons why. - overrideDerivation is not deprecated and will continue - to work, but is less nice to use and does not have as many abilities as - overrideAttrs. - - - - - Do not use this function in Nixpkgs as it evaluates a Derivation - before modifying it, which breaks package abstraction and removes - error-checking of function arguments. In addition, this - evaluation-per-function application incurs a performance penalty, - which can become a problem if many overrides are used. - It is only intended for ad-hoc customisation, such as in - ~/.nixpkgs/config.nix. - - - - - The function overrideDerivation creates a new derivation - based on an existing one by overriding the original's attributes with - the attribute set produced by the specified function. - This function is available on all - derivations defined using the makeOverridable function. - Most standard derivation-producing functions, such as - stdenv.mkDerivation, are defined using this - function, which means most packages in the nixpkgs expression, - pkgs, have this function. - - - - Example usage: - - mySed = pkgs.gnused.overrideDerivation (oldAttrs: { - name = "sed-4.2.2-pre"; - src = fetchurl { - url = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2; - sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k"; - }; - patches = []; -}); - - - - In the above example, the name, src, - and patches of the derivation will be overridden, while - all other attributes will be retained from the original derivation. - - - - The argument oldAttrs is used to refer to the attribute set of - the original derivation. - - - - A package's attributes are evaluated *before* being modified by - the overrideDerivation function. - For example, the name attribute reference - in url = "mirror://gnu/hello/${name}.tar.gz"; - is filled-in *before* the overrideDerivation function - modifies the attribute set. This means that overriding the - name attribute, in this example, *will not* change the - value of the url attribute. Instead, we need to override - both the name *and* url attributes. + Warning: this function is expensive and must not be used from within + the nixpkgs repository. - + + Example usage: + + let + pkgs = import <nixpkgs> {}; + newpkgs = pkgs.overridePackages (self: super: { + foo = super.foo.override { ... }; + }; + in ... + + + + The resulting newpkgs will have the new foo + expression, and all other expressions depending on foo will also + use the new foo expression. + + + + The behavior of this function is similar to config.packageOverrides. + + + + The self parameter refers to the final package set with the + applied overrides. Using this parameter may lead to infinite recursion if not + used consciously. + + + + The super parameter refers to the old package set. + It's equivalent to pkgs in the above example. + + + + Note that in previous versions of nixpkgs, this method replaced any changes from config.packageOverrides, + along with that from previous calls if this function was called repeatedly. + Now those previous changes will be preserved so this function can be "chained" meaningfully. + To recover the old behavior, make sure config.packageOverrides is unset, + and call this only once off a "freshly" imported nixpkgs: + + let + pkgs = import <nixpkgs> { config: {}; }; + newpkgs = pkgs.overridePackages ...; + in ... + + +
+ +
+ <pkg>.override + + + The function override is usually available for all the + derivations in the nixpkgs expression (pkgs). + + + It is used to override the arguments passed to a function. + + + Example usages: + + pkgs.foo.override { arg1 = val1; arg2 = val2; ... } + pkgs.overridePackages (self: super: { + foo = super.foo.override { barSupport = true ; }; + }) + mypkg = pkgs.callPackage ./mypkg.nix { + mydep = pkgs.mydep.override { ... }; + }) + + + + In the first example, pkgs.foo is the result of a function call + with some default arguments, usually a derivation. + Using pkgs.foo.override will call the same function with + the given new arguments. + + +
+ +
+ <pkg>.overrideAttrs + + + The function overrideAttrs allows overriding the + attribute set passed to a stdenv.mkDerivation call, + producing a new derivation based on the original one. + This function is available on all derivations produced by the + stdenv.mkDerivation function, which is most packages + in the nixpkgs expression pkgs. + + + + Example usage: + + helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec { + separateDebugInfo = true; + }); + + + + In the above example, the separateDebugInfo attribute is + overriden to be true, thus building debug info for + helloWithDebug, while all other attributes will be + retained from the original hello package. + + + + The argument oldAttrs is conventionally used to refer to + the attr set originally passed to stdenv.mkDerivation. + + + + + Note that separateDebugInfo is processed only by the + stdenv.mkDerivation function, not the generated, raw + Nix derivation. Thus, using overrideDerivation will + not work in this case, as it overrides only the attributes of the final + derivation. It is for this reason that overrideAttrs + should be preferred in (almost) all cases to + overrideDerivation, i.e. to allow using + sdenv.mkDerivation to process input arguments, as well + as the fact that it is easier to use (you can use the same attribute + names you see in your Nix code, instead of the ones generated (e.g. + buildInputs vs nativeBuildInputs, + and involves less typing. + + + +
+ + +
+ <pkg>.overrideDerivation + + + You should prefer overrideAttrs in almost all + cases, see its documentation for the reasons why. + overrideDerivation is not deprecated and will continue + to work, but is less nice to use and does not have as many abilities as + overrideAttrs. + + + + + Do not use this function in Nixpkgs as it evaluates a Derivation + before modifying it, which breaks package abstraction and removes + error-checking of function arguments. In addition, this + evaluation-per-function application incurs a performance penalty, + which can become a problem if many overrides are used. + It is only intended for ad-hoc customisation, such as in + ~/.nixpkgs/config.nix. + + + + + The function overrideDerivation creates a new derivation + based on an existing one by overriding the original's attributes with + the attribute set produced by the specified function. + This function is available on all + derivations defined using the makeOverridable function. + Most standard derivation-producing functions, such as + stdenv.mkDerivation, are defined using this + function, which means most packages in the nixpkgs expression, + pkgs, have this function. + + + + Example usage: + + mySed = pkgs.gnused.overrideDerivation (oldAttrs: { + name = "sed-4.2.2-pre"; + src = fetchurl { + url = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2; + sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k"; + }; + patches = []; + }); + + + + In the above example, the name, src, + and patches of the derivation will be overridden, while + all other attributes will be retained from the original derivation. + + + + The argument oldAttrs is used to refer to the attribute set of + the original derivation. + + + + + A package's attributes are evaluated *before* being modified by + the overrideDerivation function. + For example, the name attribute reference + in url = "mirror://gnu/hello/${name}.tar.gz"; + is filled-in *before* the overrideDerivation function + modifies the attribute set. This means that overriding the + name attribute, in this example, *will not* change the + value of the url attribute. Instead, we need to override + both the name *and* url attributes. + + + +
+ +
+ lib.makeOverridable + + + The function lib.makeOverridable is used to make the result + of a function easily customizable. This utility only makes sense for functions + that accept an argument set and return an attribute set. + + + + Example usage: + + f = { a, b }: { result = a+b; } + c = lib.makeOverridable f { a = 1; b = 2; } + + + + + The variable c is the value of the f function + applied with some default arguments. Hence the value of c.result + is 3, in this example. + + + + The variable c however also has some additional functions, like + c.override which can be used to + override the default arguments. In this example the value of + (c.override { a = 4; }).result is 6. + + +
-
- lib.makeOverridable +
+ Generators - The function lib.makeOverridable is used to make the result - of a function easily customizable. This utility only makes sense for functions - that accept an argument set and return an attribute set. + Generators are functions that create file formats from nix + data structures, e. g. for configuration files. + There are generators available for: INI, + JSON and YAML - Example usage: - - f = { a, b }: { result = a+b; } -c = lib.makeOverridable f { a = 1; b = 2; } - + All generators follow a similar call interface: generatorName + configFunctions data, where configFunctions is a + set of user-defined functions that format variable parts of the content. + They each have common defaults, so often they do not need to be set + manually. An example is mkSectionName ? (name: libStr.escape [ "[" "]" + ] name) from the INI generator. It gets the name + of a section and returns a sanitized name. The default + mkSectionName escapes [ and + ] with a backslash. - - The variable c is the value of the f function - applied with some default arguments. Hence the value of c.result - is 3, in this example. - + Nix store paths can be converted to strings by enclosing a + derivation attribute like so: "${drv}". - The variable c however also has some additional functions, like - c.override which can be used to - override the default arguments. In this example the value of - (c.override { a = 4; }).result is 6. + Detailed documentation for each generator can be found in + lib/generators.nix.
@@ -370,37 +413,37 @@ c = lib.makeOverridable f { a = 1; b = 2; }
- pkgs.dockerTools +pkgs.dockerTools - + pkgs.dockerTools is a set of functions for creating and manipulating Docker images according to the - Docker Image Specification v1.0.0 + Docker Image Specification v1.0.0 . Docker itself is not used to perform any of the operations done by these functions. - + - + - The dockerTools API is unstable and may be subject to - backwards-incompatible changes in the future. + The dockerTools API is unstable and may be subject to + backwards-incompatible changes in the future. - + -
+
buildImage - This function is analogous to the docker build command, - in that can used to build a Docker-compatible repository tarball containing - a single image with one or multiple layers. As such, the result - is suitable for being loaded in Docker with docker load. + This function is analogous to the docker build command, + in that can used to build a Docker-compatible repository tarball containing + a single image with one or multiple layers. As such, the result + is suitable for being loaded in Docker with docker load. - The parameters of buildImage with relative example values are - described below: + The parameters of buildImage with relative example values are + described below: Docker build @@ -408,11 +451,11 @@ c = lib.makeOverridable f { a = 1; b = 2; } buildImage { name = "redis"; tag = "latest"; - + fromImage = someBaseImage; fromImageName = null; fromImageTag = "latest"; - + contents = pkgs.redis; runAsRoot = '' #!${stdenv.shell} @@ -431,131 +474,131 @@ c = lib.makeOverridable f { a = 1; b = 2; } The above example will build a Docker image redis/latest - from the given base image. Loading and running this image in Docker results in - redis-server being started automatically. + from the given base image. Loading and running this image in Docker results in + redis-server being started automatically. - + - name specifies the name of the resulting image. - This is the only required argument for buildImage. + name specifies the name of the resulting image. + This is the only required argument for buildImage. - + - + - tag specifies the tag of the resulting image. - By default it's latest. + tag specifies the tag of the resulting image. + By default it's latest. - + - + - fromImage is the repository tarball containing the base image. - It must be a valid Docker image, such as exported by docker save. - By default it's null, which can be seen as equivalent - to FROM scratch of a Dockerfile. + fromImage is the repository tarball containing the base image. + It must be a valid Docker image, such as exported by docker save. + By default it's null, which can be seen as equivalent + to FROM scratch of a Dockerfile. - - - - - fromImageName can be used to further specify - the base image within the repository, in case it contains multiple images. - By default it's null, in which case - buildImage will peek the first image available - in the repository. - - + - + - fromImageTag can be used to further specify the tag - of the base image within the repository, in case an image contains multiple tags. - By default it's null, in which case - buildImage will peek the first tag available for the base image. + fromImageName can be used to further specify + the base image within the repository, in case it contains multiple images. + By default it's null, in which case + buildImage will peek the first image available + in the repository. - + - + - contents is a derivation that will be copied in the new - layer of the resulting image. This can be similarly seen as - ADD contents/ / in a Dockerfile. - By default it's null. + fromImageTag can be used to further specify the tag + of the base image within the repository, in case an image contains multiple tags. + By default it's null, in which case + buildImage will peek the first tag available for the base image. - + - + - runAsRoot is a bash script that will run as root - in an environment that overlays the existing layers of the base image with - the new resulting layer, including the previously copied - contents derivation. - This can be similarly seen as - RUN ... in a Dockerfile. - - + contents is a derivation that will be copied in the new + layer of the resulting image. This can be similarly seen as + ADD contents/ / in a Dockerfile. + By default it's null. + + + + + + runAsRoot is a bash script that will run as root + in an environment that overlays the existing layers of the base image with + the new resulting layer, including the previously copied + contents derivation. + This can be similarly seen as + RUN ... in a Dockerfile. + + - Using this parameter requires the kvm - device to be available. + Using this parameter requires the kvm + device to be available. - + - + - + - config is used to specify the configuration of the - containers that will be started off the built image in Docker. - The available options are listed in the - + config is used to specify the configuration of the + containers that will be started off the built image in Docker. + The available options are listed in the + Docker Image Specification v1.0.0 - . + . - + - After the new layer has been created, its closure - (to which contents, config and - runAsRoot contribute) will be copied in the layer itself. - Only new dependencies that are not already in the existing layers will be copied. + After the new layer has been created, its closure + (to which contents, config and + runAsRoot contribute) will be copied in the layer itself. + Only new dependencies that are not already in the existing layers will be copied. - At the end of the process, only one new single layer will be produced and - added to the resulting image. + At the end of the process, only one new single layer will be produced and + added to the resulting image. - The resulting repository will only list the single image - image/tag. In the case of - it would be redis/latest. + The resulting repository will only list the single image + image/tag. In the case of + it would be redis/latest. - It is possible to inspect the arguments with which an image was built - using its buildArgs attribute. + It is possible to inspect the arguments with which an image was built + using its buildArgs attribute. -
+
-
+
pullImage - This function is analogous to the docker pull command, - in that can be used to fetch a Docker image from a Docker registry. - Currently only registry v1 is supported. - By default Docker Hub - is used to pull images. + This function is analogous to the docker pull command, + in that can be used to fetch a Docker image from a Docker registry. + Currently only registry v1 is supported. + By default Docker Hub + is used to pull images. - Its parameters are described in the example below: + Its parameters are described in the example below: Docker pull @@ -573,73 +616,73 @@ c = lib.makeOverridable f { a = 1; b = 2; } - + - imageName specifies the name of the image to be downloaded, - which can also include the registry namespace (e.g. library/debian). - This argument is required. + imageName specifies the name of the image to be downloaded, + which can also include the registry namespace (e.g. library/debian). + This argument is required. - - - - - imageTag specifies the tag of the image to be downloaded. - By default it's latest. - - + - + - imageId, if specified this exact image will be fetched, instead - of imageName/imageTag. However, the resulting repository - will still be named imageName/imageTag. - By default it's null. + imageTag specifies the tag of the image to be downloaded. + By default it's latest. - + - + - sha256 is the checksum of the whole fetched image. - This argument is required. + imageId, if specified this exact image will be fetched, instead + of imageName/imageTag. However, the resulting repository + will still be named imageName/imageTag. + By default it's null. + + + + + + sha256 is the checksum of the whole fetched image. + This argument is required. - The checksum is computed on the unpacked directory, not on the final tarball. + The checksum is computed on the unpacked directory, not on the final tarball. - + - + - In the above example the default values are shown for the variables - indexUrl and registryVersion. - Hence by default the Docker.io registry is used to pull the images. + In the above example the default values are shown for the variables + indexUrl and registryVersion. + Hence by default the Docker.io registry is used to pull the images. - + - -
- -
+ +
+ +
exportImage - This function is analogous to the docker export command, - in that can used to flatten a Docker image that contains multiple layers. - It is in fact the result of the merge of all the layers of the image. - As such, the result is suitable for being imported in Docker - with docker import. + This function is analogous to the docker export command, + in that can used to flatten a Docker image that contains multiple layers. + It is in fact the result of the merge of all the layers of the image. + As such, the result is suitable for being imported in Docker + with docker import. - + Using this function requires the kvm device to be available. - + - The parameters of exportImage are the following: + The parameters of exportImage are the following: Docker export @@ -648,35 +691,35 @@ c = lib.makeOverridable f { a = 1; b = 2; } fromImage = someLayeredImage; fromImageName = null; fromImageTag = null; - + name = someLayeredImage.name; } - The parameters relative to the base image have the same synopsis as - described in , except that - fromImage is the only required argument in this case. + The parameters relative to the base image have the same synopsis as + described in , except that + fromImage is the only required argument in this case. - The name argument is the name of the derivation output, - which defaults to fromImage.name. + The name argument is the name of the derivation output, + which defaults to fromImage.name. -
+
-
+
shadowSetup - This constant string is a helper for setting up the base files for managing - users and groups, only if such files don't exist already. - It is suitable for being used in a - runAsRoot script for cases like - in the example below: + This constant string is a helper for setting up the base files for managing + users and groups, only if such files don't exist already. + It is suitable for being used in a + runAsRoot script for cases like + in the example below: - + Shadow base files buildImage { @@ -695,13 +738,13 @@ c = lib.makeOverridable f { a = 1; b = 2; } - Creating base files like /etc/passwd or - /etc/login.defs are necessary for shadow-utils to - manipulate users and groups. + Creating base files like /etc/passwd or + /etc/login.defs are necessary for shadow-utils to + manipulate users and groups. - -
- + +
+
diff --git a/lib/generators.nix b/lib/generators.nix index a1396873695..27d4142e8d9 100644 --- a/lib/generators.nix +++ b/lib/generators.nix @@ -1,6 +1,11 @@ -/* Functions that generate widespread config file +/* Functions that generate widespread file * formats from nix data structures. + * + * They all follow a similar interface: + * generator { config-attrs } data + * * Tests can be found in ./tests.nix + * Documentation in the manual, #sec-generators */ with import ./trivial.nix; let