Commit graph

42 commits

Author SHA1 Message Date
Matthew Bauer 0046802ab6 fetchurl: only allow empty hash when cacert is available
We can use cacert to validate that the data passes SSL certificates.
Normally, this doesn’t happen because we already have the hash, but in
the hash = "" case we don’t.
2020-06-09 12:56:27 -05:00
Matthew Bauer 6367a55312 fetchurl: don’t run ‘postFetch’ on hashed-mirrors
hashed-mirrors are content addressed. So if $outputHash is in the
hashed-mirror, changes from ‘postFetch’ would already be made. So,
running postFetch will end up applying the change /again/, which we
don’t want.
2020-04-16 16:36:56 -05:00
Yegor Timoshenko 6fad2ae94b
fetchurl: warn on rev archives, resolves #32999 2018-10-16 13:46:19 +00:00
Yegor Timoshenko a26d9e3ba8
Readd --retry 3 2018-06-11 01:48:00 +00:00
Dan Peebles 0cb623c3d9
fetchurl: add user agent
It would be nice to be able to track Nix requests. It's not trustworthy,
but can be helpful for stats and routing in HTTP logs.

Since `fetchurl` is used so widely, we should "magically" get a UA on
`fetchzip`, `fetchFromGitHub`, and other related fetchers.

Since `fetchurl` is only used for fixed-output derivations, this should
cause no mass rebuild.

User-Agent example: curl/7.57.0 Nixpkgs/18.03
2017-12-23 22:20:56 +00:00
Thomas Tuegel c25199f697
fetchurl: remove unpaired call to `stopNest'
Fixes #27406.

Commit 5d4efb2c81 added an assertion to `stopNest'
which requires it be correctly paired with `startNest'. `fetchurl' calls
`stopNest', but never calls `startNest'; the former calls are removed.
2017-07-21 15:45:29 -05:00
Domen Kožar c7383cb34b fetchurl: support executables 2016-01-19 11:17:49 +01:00
Eelco Dolstra bb672805cd fetchurl: Always use tarballs.nixos.org
Otherwise, if the upstream mirror changes (rather than deletes) a
file, then tarballs.nixos.org won't be used even if it has a copy of
the original file, and so we'll get a hash mismatch.
2015-12-10 16:06:12 +01:00
ts468 db25ece3c8 fetchurl/builder.sh: style fix, close #10260 2015-10-07 09:48:58 +02:00
Aristid Breitkreuz 5c0816567d fix fetchurl 2014-10-05 19:23:27 +02:00
Domen Kožar d6b7778753 fetchurl: fix mirroring after ecf5a08831 2014-10-05 09:53:18 +02:00
Domen Kožar fd48e801d2 fetchurl: resume download of curl exits with status code 18 2014-10-01 13:58:42 +02:00
Eelco Dolstra c8df888858 Add a function "fetchzip"
This function downloads and unpacks a file in one fixed-output
derivation. This is primarily useful for dynamically generated zip
files, such as GitHub's /archive URLs, where the unpacked content of
the zip file doesn't change, but the zip file itself may (e.g. due to
minor changes in the compression algorithm, or changes in timestamps).

Fetchzip is implemented by extending fetchurl with a "postFetch" hook
that is executed after the file has been downloaded. This hook can
thus perform arbitrary checks or transformations on the downloaded
file.
2014-05-08 15:30:17 +02:00
Gergely Risko 07b26ce493 fetchUrl: add curlOpts parameter (merge #535) 2013-05-16 10:45:16 +02:00
Eelco Dolstra 04765da09a fetchurl: Don't wait too long for the hashed mirrors
If the hashed mirror (nixos.org/tarballs) doesn't respond in 15
seconds, give up and try the original URL.
2013-02-06 15:15:28 +01:00
Eelco Dolstra f337ab4430 fetchurl: Remove some compatibility hacks 2013-02-06 15:00:33 +01:00
Eelco Dolstra 4b50191fd4 * Remove a not very useful message that didn't get printed anyway
because the test condition was wrong.

svn path=/nixpkgs/trunk/; revision=26147
2011-03-03 14:59:58 +00:00
Yury G. Kudryashov c74d23dbdf Fix typo, thanks to Rob Vermaas
svn path=/nixpkgs/trunk/; revision=22031
2010-05-28 06:49:32 +00:00
Yury G. Kudryashov 952bc80b4d Let fetchurl retry failed downloads. Add NIX_CURL_FLAGS impure var
svn path=/nixpkgs/trunk/; revision=22006
2010-05-27 18:59:19 +00:00
Marc Weber ec0e356b9a Revert "adding NIX_CONTINUE_DOWNLOADS feature, see mkdir comment"
This reverts commit -r16401

svn path=/nixpkgs/trunk/; revision=16405
2009-07-16 17:17:23 +00:00
Marc Weber 10f6c2592c adding NIX_CONTINUE_DOWNLOADS feature, see mkdir comment
svn path=/nixpkgs/trunk/; revision=16401
2009-07-16 15:18:26 +00:00
Michael Raskin f067828cdc Allow question marks in URL...
svn path=/nixpkgs/trunk/; revision=15507
2009-05-08 17:00:33 +00:00
Eelco Dolstra aa84bce64d * Get rid of all references to nix.cs.uu.nl.
svn path=/nixpkgs/trunk/; revision=13299
2008-11-14 16:57:19 +00:00
Eelco Dolstra eebb241519 * Write the list of mirrors to a file that we can reuse between
fetchurl instantiations, instead of passing the mirrors to fetchurl
  instantiations via environment variables.  This makes the resulting
  store derivations (.drv files) much smaller, which in turn makes
  nix-env/nix-instantiate faster (4.8 -> 4.2 seconds on nix-env -qa
  --out-path).

svn path=/nixpkgs/trunk/; revision=12695
2008-08-22 15:53:21 +00:00
Eelco Dolstra a7c209e474 * Interface to support mirror:// in nix-prefetch-url. The function
resolveMirrorURLs produces a file containing the URLs to which a
  mirror:// URL resolves.

svn path=/nixpkgs/trunk/; revision=12410
2008-07-23 16:04:10 +00:00
Eelco Dolstra 76a8d120a4 * Allow the user to override the list of mirrors for specific
mirror:// sites through environment variables, e.g.

  NIX_MIRRORS_gnu="ftp://ftp.nluug.nl/pub/gnu/ ftp://ftp.gnu.org/pub/gnu/"

  or

  NIX_MIRRORS_sourceforge="http://surfnet.dl.sourceforge.net/sourceforge/"

svn path=/nixpkgs/trunk/; revision=9302
2007-09-11 15:00:49 +00:00
Eelco Dolstra 6c4fd2e3df * Allow the user to override the list of content-addressable mirrors
from the command-line by setting the NIX_HASHED_MIRRORS environment
  variable.

svn path=/nixpkgs/trunk/; revision=9301
2007-09-11 13:48:53 +00:00
Eelco Dolstra 45a2c87402 * Support for mirror:// URLs a la Gentoo (NIXPKGS-70). Instead of
fetchurl {
      url = http://heanet.dl.sourceforge.net/sourceforge/zapping/zapping-0.9.6.tar.bz2;
      md5 = "8306775c6a11de4d72345b5eee970ea6";
    };

  you can write

    fetchurl {
      url = mirror://sourceforge/zapping/zapping-0.9.6.tar.bz2;
      md5 = "8306775c6a11de4d72345b5eee970ea6";
    };

  which causes fetchurl to try the SourceForge mirrors listed in the
  `sourceforge' attribute in build-support/fetchurl/mirrors.nix.
  (They're currently tried in sequence, and the lists of mirrors are
  not configurable yet.)

  The syntax for mirror URLs is mirror://site/path/to/file, where
  `site' is currently one of `sourceforge', `gnu' (mirrors of
  ftp://ftp.gnu.org/pub/gnu) and `kernel' (mirrors of
  http://www.all.kernel.org/pub/).

svn path=/nixpkgs/trunk/; revision=9197
2007-08-27 12:44:01 +00:00
Eelco Dolstra 50321b735b * Basic transparent mirror support in fetchurl (NIXPKGS-70). When
fetching a file with hash HASH of type TYPE, we first try to
  download <base-url>/<type>/<hash>, where <base-url> is one of a list
  of mirrors.  For instance, given

    src = fetchurl {
      url = http://releases.mozilla.org/pub/mozilla.org/firefox/releases/2.0.0.6/source/firefox-2.0.0.6-source.tar.bz2;
      sha1 = "eb72f55e4a8bf08e8c6ef227c0ade3d068ba1082";
    };

  and the mirror list [http://nix.cs.uu.nl/dist/tarballs], we first
  try to download

    eb72f55e4a

  and if that fails, we use the original URL.

  The list of mirrors is not yet user-configurable.

* `fetchurl' now also accepts an argument `urls' instead of `url' for
  a list of alternative download locations, which fetchurl will try in
  sequence.

svn path=/nixpkgs/trunk/; revision=9190
2007-08-23 15:22:30 +00:00
Eelco Dolstra f59f5d7d96 * Handle weird cases when the server redirects us while setting a
cookie.

svn path=/nixpkgs/trunk/; revision=7352
2006-12-15 21:28:59 +00:00
Eelco Dolstra f95a93197a * Don't use EPSV.
svn path=/nixpkgs/trunk/; revision=6894
2006-10-28 22:07:25 +00:00
Eelco Dolstra 82e678362f * "." -> "source".
svn path=/nixpkgs/trunk/; revision=4335
2005-12-05 14:11:09 +00:00
Eelco Dolstra 44a1f60787 * md5' -> id' in compatibility mode.
svn path=/nixpkgs/trunk/; revision=2273
2005-02-22 15:34:58 +00:00
Eelco Dolstra 4eba4a211e * Check the hash, but only on Nix <= 0.7.
svn path=/nixpkgs/trunk/; revision=2272
2005-02-22 15:23:56 +00:00
Eelco Dolstra cdaecff49f * Use Nix 0.8-style fixed outputs.
svn path=/nixpkgs/trunk/; revision=2257
2005-02-21 15:52:37 +00:00
Eelco Dolstra 7bf54b734e * Follow redirects.
svn path=/nixpkgs/trunk/; revision=899
2004-04-01 19:10:37 +00:00
Eelco Dolstra ab7af4709a * Add `--fail' to curl invocation to catch errors.
svn path=/nixpkgs/trunk/; revision=889
2004-04-01 12:23:02 +00:00
Eelco Dolstra 12ae5363ea * Remove trivial builders.
* Make builders unexecutable by removing the hash-bang line and
  execute permission.
* Convert calls to `derivation' to `mkDerivation'.
* Remove `system' and `stdenv' attributes from calls to
  `mkDerivation'.  These transformations were all done automatically,
  so it is quite possible I broke stuff.
* Put the `mkDerivation' function in stdenv/generic.

svn path=/nixpkgs/trunk/; revision=874
2004-03-29 17:23:01 +00:00
Eelco Dolstra 4df6ec860c * Typo fix.
svn path=/nixpkgs/trunk/; revision=850
2004-03-27 15:44:05 +00:00
Eelco Dolstra 5941f66f0e * The stdenv setup script now defines a generic builder that allows
builders for typical Autoconf-style to be much shorten, e.g.,

    . $stdenv/setup
    genericBuild

  The generic builder does lots of stuff automatically:

  - Unpacks source archives specified by $src or $srcs (it knows about
    gzip, bzip2, tar, zip, and unpacked source trees).
  - Determines the source tree.
  - Applies patches specified by $patches.
  - Fixes libtool not to search for libraries in /lib etc.
  - Runs `configure'.
  - Runs `make'.
  - Runs `make install'.
  - Strips debug information from static libraries.
  - Writes nested log information (in the format accepted by
    `log2xml').

  There are also lots of hooks and variables to customise the generic
  builder.  See `stdenv/generic/docs.txt'.

* Adapted the base packages (i.e., the ones used by stdenv) to use the
  generic builder.

* We now use `curl' instead of `wget' to download files in `fetchurl'.

* Neither `curl' nor `wget' are part of stdenv.  We shouldn't
  encourage people to download stuff in builders (impure!).

* Updated some packages.

* `buildinputs' is now `buildInputs' (but the old name also works).

* `findInputs' in the setup script now prevents inputs from being
  processed multiple times (which could happen, e.g., if an input was
  a propagated input of several other inputs; this caused the size
  variables like $PATH to blow up exponentially in the worst case).

* Patched GNU Make to write nested log information in the format
  accepted by `log2xml'.  Also, prior to writing the build command,
  Make now writes a line `building X' to indicate what is being
  built.  This is unfortunately often obscured by the gigantic tool
  invocations in many Makefiles.  The actual build commands are marked
  `unimportant' so that they don't clutter pages generated by
  `log2html'.


svn path=/nixpkgs/trunk/; revision=845
2004-03-19 16:53:04 +00:00
Eelco Dolstra 38bc744a0b * Use md5sum instead of nix-hash.
svn path=/nixpkgs/trunk/; revision=584
2003-12-21 16:00:17 +00:00
Eelco Dolstra 48c3faca51 * Renamed pkgs-ng to pkgs.
svn path=/nixpkgs/trunk/; revision=502
2003-11-14 09:59:13 +00:00
Renamed from pkgs-ng/build-support/fetchurl/builder.sh (Browse further)