nixpkgs/pkgs/build-support
Graham Christensen fd045173ce referencesByPopularity: init to sort packages by a cachability heuristic
Using a simple algorithm, convert the references to a path in to a
sorted list of dependent paths based on how often they're referenced
and how deep in the tree they live. Equally-"popular" paths are then
sorted by name.

The existing writeReferencesToFile prints the paths in a simple
ascii-based sorting of the paths.

Sorting the paths by graph improves the chances that the difference
between two builds appear near the end of the list, instead of near
the beginning. This makes a difference for Nix builds which export a
closure for another program to consume, if that program implements its
own level of binary diffing.

For an example, Docker Images. If each store path is a separate layer
then Docker Images can be very efficiently transfered between systems,
and we get very good cache reuse between images built with the same
version of Nixpkgs. However, since Docker only reliably supports a
small number of layers (42) it is important to pick the individual
layers carefully. By storing very popular store paths in the first 40
layers, we improve the chances that the next Docker image will share
many of those layers.*

Given the dependency tree:

    A - B - C - D -\
     \   \   \      \
      \   \   \      \
       \   \ - E ---- F
        \- G

Nodes which have multiple references are duplicated:

    A - B - C - D - F
     \   \   \
      \   \   \- E - F
       \   \
        \   \- E - F
         \
          \- G

Each leaf node is now replaced by a counter defaulted to 1:

    A - B - C - D - (F:1)
     \   \   \
      \   \   \- E - (F:1)
       \   \
        \   \- E - (F:1)
         \
          \- (G:1)

Then each leaf counter is merged with its parent node, replacing the
parent node with a counter of 1, and each existing counter being
incremented by 1. That is to say `- D - (F:1)` becomes `- (D:1, F:2)`:

    A - B - C - (D:1, F:2)
     \   \   \
      \   \   \- (E:1, F:2)
       \   \
        \   \- (E:1, F:2)
         \
          \- (G:1)

Then each leaf counter is merged with its parent node again, merging
any counters, then incrementing each:

    A - B - (C:1, D:2, E:2, F:5)
     \   \
      \   \- (E:1, F:2)
       \
        \- (G:1)

And again:

    A - (B:1, C:2, D:3, E:4, F:8)
     \
      \- (G:1)

And again:

    (A:1, B:2, C:3, D:4, E:5, F:9, G:2)

and then paths have the following "popularity":

    A     1
    B     2
    C     3
    D     4
    E     5
    F     9
    G     2

and the popularity contest would result in the paths being printed as:

    F
    E
    D
    C
    B
    G
    A

* Note: People who have used a Dockerfile before assume Docker's
Layers are inherently ordered. However, this is not true -- Docker
layers are content-addressable and are not explicitly layered until
they are composed in to an Image.
2018-09-26 15:50:10 -04:00
..
agda
bintools-wrapper Revert "Merge pull request #44767 from obsidiansystems/wrapper-env-var-path" 2018-08-22 01:14:53 +02:00
build-bazel-package bazel-deps: init at 2018-05-31 (#43018) 2018-07-09 22:38:45 +00:00
build-dotnet-package treewide: Manual fix more pkg-config build-inputs 2017-09-21 15:49:54 -04:00
build-fhs-userenv reewide: Purge all uses stdenv.system and top-level system 2018-08-30 17:20:32 -04:00
build-setupcfg Revert "buildSetupcfg: Include unzip for zip sources." 2018-04-14 20:12:22 -04:00
buildenv Merge pull request #27780 from tilpner/overridable-buildenv 2017-09-14 17:26:33 +02:00
cc-wrapper treewide: cleanup some references to bash 2018-09-04 22:05:02 +00:00
docker dockertools: tarsum: turn in to a buildInput 2018-09-26 15:50:04 -04:00
dotnetbuildhelpers
dotnetenv [bot]: remove unreferenced code 2018-07-20 18:48:37 +00:00
emacs Fix pname to ename propagation if null 2018-07-27 08:15:46 +02:00
expand-response-params expand-response-params: Build more normally 2017-08-25 15:10:02 -04:00
fetchbower Merge remote-tracking branch 'upstream/master' into staging 2018-01-14 21:18:27 +02:00
fetchbzr treewide: Fetchers should use stdenvNoCC. 2018-01-10 11:18:44 -05:00
fetchcvs bsd: init netbsd & openbsd userland 2018-04-24 14:16:35 -05:00
fetchdarcs pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
fetchdocker Merge branch 'no-toPath' 2018-09-06 08:09:53 -04:00
fetchegg treewide: Fetchers should use stdenvNoCC. 2018-01-10 11:18:44 -05:00
fetchfossil treewide: Fixed output fetch* derivations should use nativeBuildInputs 2018-01-09 20:14:46 -05:00
fetchgit pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
fetchgitlocal pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
fetchgx Merge remote-tracking branch 'upstream/master' into staging 2018-01-14 21:18:27 +02:00
fetchhg pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
fetchipfs treewide: Fixed output fetch* derivations should use nativeBuildInputs 2018-01-09 20:14:46 -05:00
fetchmavenartifact
fetchmtn treewide: Fetchers should use stdenvNoCC. 2018-01-10 11:18:44 -05:00
fetchnuget pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
fetchpatch fetchpatch: patchutils -> buildPackages.patchutils 2018-08-21 17:21:25 -04:00
fetchrepoproject treewide: Fetchers should use stdenvNoCC. 2018-01-10 11:18:44 -05:00
fetchs3 pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
fetchsvn treewide: Fetchers should use stdenvNoCC. 2018-01-10 11:18:44 -05:00
fetchsvnrevision
fetchsvnssh treewide: Fetchers should use stdenvNoCC. 2018-01-10 11:18:44 -05:00
fetchurl fetchurl: add sageupstream mirror 2018-06-30 01:20:44 +02:00
fetchzip fetchzip: Use unzip from buildPackages 2018-09-21 12:55:32 -04:00
icon-conv-tools
kernel treewide: Remove usage of remaining redundant platform compatability stuff 2018-08-30 17:20:32 -04:00
libredirect libredirect: add description 2017-12-14 19:03:35 +00:00
make-desktopitem make-desktopitem: make genericName optional 2017-06-05 21:55:23 -05:00
make-startupitem make-autostartitem: Use the right path for storing the desktop files 2017-07-06 00:44:41 +02:00
make-symlinks
mkshell mkShell: add builder (#30975) 2017-12-20 23:42:07 +00:00
mono-dll-fixer
nix-prefetch-github nix-prefetch-github: init -> 1.3 2018-05-22 20:55:07 +02:00
nuke-references
ocaml
references-by-popularity referencesByPopularity: init to sort packages by a cachability heuristic 2018-09-26 15:50:10 -04:00
release reewide: Purge all uses stdenv.system and top-level system 2018-08-30 17:20:32 -04:00
remove-references-to Add removeReferencesTo for removing specific refs 2017-03-09 12:01:55 +00:00
rust buildRustCrate: added some edge cases with binaries 2018-09-13 22:00:29 +02:00
setup-hooks makeWrapper: document --set-default 2018-08-24 19:46:16 +02:00
singularity-tools pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
skaware skaware: switch from git repos to tarballs 2018-09-06 11:53:22 +02:00
src-only
substitute
substitute-files
templaterpm
upstream-updater
vm virtualization/qemu-vm: fix and improve virtio/scsi switching 2018-09-22 23:29:19 +02:00
wrapper-common {bintools,cc}-wrapper: Factor out role accumulation logic 2018-05-07 15:10:45 -04:00
build-maven.nix buildMaven: Check for authenticated attribute 2018-07-25 16:00:52 +02:00
build-pecl.nix Revert "fix phpPackages memcache,memcached,xdebug" 2017-12-10 12:12:43 +01:00
closure-info.nix Cleanup 2018-02-27 19:59:26 +01:00
dhall-to-nix.nix
plugins.nix Ultrastar (#26524) 2017-06-14 11:29:31 +02:00
replace-dependency.nix
setup-systemd-units.nix Add setupSystemdUnits function. 2017-03-24 15:47:51 -04:00
source-from-head-fun.nix
trivial-builders.nix lib: ensure directories of linkFarm links exist (#45628) 2018-09-01 14:53:23 +02:00