Commit graph

67 commits

Author SHA1 Message Date
Vladimír Čunát 7c879d342d Merge #10816: improve FreeBSD support 2016-01-05 09:50:10 +01:00
Charles Strahan b6c06e216b ruby: new bundler infrastructure
This improves our Bundler integration (i.e. `bundlerEnv`).

Before describing the implementation differences, I'd like to point a
breaking change: buildRubyGem now expects `gemName` and `version` as
arguments, rather than a `name` attribute in the form of
"<gem-name>-<version>".

Now for the differences in implementation.

The previous implementation installed all gems at once in a single
derivation. This was made possible by using a set of monkey-patches to
prevent Bundler from downloading gems impurely, and to help Bundler
find and activate all required gems prior to installation. This had
several downsides:

* The patches were really hard to understand, and required subtle
  interaction with the rest of the build environment.
* A single install failure would cause the entire derivation to fail.

The new implementation takes a different approach: we install gems into
separate derivations, and then present Bundler with a symlink forest
thereof. This has a couple benefits over the existing approach:

* Fewer patches are required, with less interplay with the rest of the
  build environment.
* Changes to one gem no longer cause a rebuild of the entire dependency
  graph.
* Builds take 20% less time (using gitlab as a reference).

It's unfortunate that we still have to muck with Bundler's internals,
though it's unavoidable with the way that Bundler is currently designed.
There are a number improvements that could be made in Bundler that would
simplify our packaging story:

* Bundler requires all installed gems reside within the same prefix
  (GEM_HOME), unlike RubyGems which allows for multiple prefixes to
  be specified through GEM_PATH. It would be ideal if Bundler allowed
  for packages to be installed and sourced from multiple prefixes.
* Bundler installs git sources very differently from how RubyGems
  installs gem packages, and, unlike RubyGems, it doesn't provide a
  public interface (CLI or programmatic) to guide the installation of a
  single gem. We are presented with the options of either
  reimplementing a considerable portion Bundler, or patch and use parts
  of its internals; I choose the latter. Ideally, there would be a way
  to install gems from git sources in a manner similar to how we drive
  `gem` to install gem packages.
* When a bundled program is executed (via `bundle exec` or a
  binstub that does `require 'bundler/setup'`), the setup process reads
  the Gemfile.lock, activates the dependencies, re-serializes the lock
  file it read earlier, and then attempts to overwrite the Gemfile.lock
  if the contents aren't bit-identical. I think the reasoning is that
  by merely running an application with a newer version of Bundler, you'll
  automatically keep the Gemfile.lock up-to-date with any changes in the
  format. Unfortunately, that doesn't play well with any form of
  packaging, because bundler will immediately cause the application to
  abort when it attempts to write to the read-only Gemfile.lock in the
  store. We work around this by normalizing the Gemfile.lock with the
  version of Bundler that we'll use at runtime before we copy it into
  the store. This feels fragile, but it's the best we can do without
  changes upstream, or resorting to more delicate hacks.

With all of the challenges in using Bundler, one might wonder why we
can't just cut Bundler out of the picture and use RubyGems. After all,
Nix provides most of the isolation that Bundler is used for anyway.

The problem, however, is that almost every Rails application calls
`Bundler::require` at startup (by way of the default project templates).
Because bundler will then, by default, `require` each gem listed in the
Gemfile, Rails applications are almost always written such that none of
the source files explicitly require their dependencies. That leaves us
with two options: support and use Bundler, or maintain massive patches
for every Rails application that we package.

Closes #8612
2015-12-29 09:30:21 -05:00
Domen Kožar 4e4931507b fix mesos build, fixes #11638 2015-12-28 22:28:38 +01:00
Domen Kožar d83a97823c buildPythonPackage: fix a few more wheel packages 2015-11-21 21:44:12 +01:00
Domen Kožar 704c8bab41 buildPythonPackage: fix standalone applications using it 2015-11-21 21:44:12 +01:00
Pascal Wittmann 7fd2796e99 Replace 'with plaforms; platform' with 'platforms.platform' 2015-11-17 21:30:43 +01:00
William A. Kennington III 0a7176c5a1 Revert "bundlerEnv: Use released versions of bundler"
This reverts commit 7bc8b1561e.

This breaks some ruby gems and needs to be reworked again.
2015-09-20 22:05:29 -07:00
William A. Kennington III 7bc8b1561e bundlerEnv: Use released versions of bundler
We were using HEAD for unreleased features. These features are now in
release builds so we should go back to using those. This also means we
won't have to deal with hash mismatches for all ruby packages.
2015-09-20 18:56:56 -07:00
Domen Kožar 0b4ec468c5 disable chronos /cc @offlinehacker
(cherry picked from commit 83cc494542b9c53921616caa06984bd72f4e3dea)
Signed-off-by: Domen Kožar <domen@dev.si>
2015-09-15 05:41:20 +02:00
Domen Kožar c0e97bb547 panamax_ui: fix libv8 pinpoint 2015-09-03 10:30:11 +02:00
Eelco Dolstra 8fc039188e Mark some packages with undownloadable source as broken 2015-09-01 12:00:04 +02:00
Jaka Hudoklin 559e2ab951 kubernetes: 0.18.0 -> 1.0.3 2015-08-19 23:51:54 +02:00
Pascal Wittmann 590e11bc53 mesos: fix build by applying patch from review.apache.org 2015-08-07 18:14:18 +02:00
Charles Strahan bc042fe0ff mesos: 0.22.1 -> 0.23.0 2015-08-04 06:11:32 -04:00
Eelco Dolstra 55932c1bec Don't statically depend on cacert for certificates
This reverts commit cd52c04456 and
others.

Managing certificates (including revoking certificates and adding
custom certificates) becomes extremely painful if every package in the
system potentially depends on a different copy of cacert. Also, it
makes updating cacert rather expensive.
2015-07-31 01:34:58 +02:00
William A. Kennington III 333f145d76 Merge branch 'master.upstream' into staging.upstream 2015-07-13 15:11:31 -07:00
Edward Tjörnhammar 036722ab7c spark: 1.1.1 -> 1.4.0 2015-07-10 01:36:47 +02:00
William A. Kennington III dae5b35463 ruby: Fixup libv8 versions 2015-07-08 15:10:50 -07:00
William A. Kennington III 7cda24c01e Merge branch 'master.upstream' into staging.upstream 2015-07-08 14:11:16 -07:00
Jaka Hudoklin 4e33dd4e99 openshift: init at 1.0.1 2015-07-08 12:18:06 +02:00
Eelco Dolstra 4a57b07653 Merge remote-tracking branch 'origin/master' into staging 2015-06-11 00:18:24 +02:00
Jaka Hudoklin 392fc849e4 Update kubernetes to 0.18.0, fix module 2015-06-08 13:10:32 +02:00
William A. Kennington III 9d6555dc0a Merge branch 'master.upstream' into staging.upstream 2015-06-06 12:04:42 -07:00
William A. Kennington III ffd0539eba cacert: store ca-bundle.crt in $out/etc/ssl/certs instead of $out 2015-06-05 13:00:52 -07:00
William A. Kennington III 867d2c5c46 openssl: Remove References to OPENSSL_X509_CERT_FILE 2015-05-31 15:50:51 -07:00
William A. Kennington III 14c1e0fa1f Fix ca-bundle paths 2015-05-29 14:03:34 -07:00
rushmorem 6771eefd43 Update Mesos Library environment variable 2015-05-15 19:27:00 +02:00
Domen Kožar f5edc27298 Merge pull request #7844 from rushmorem/mesos-0.22-fix
Fix mesos-0.22 build and update to 0.22.1
2015-05-14 20:00:19 +02:00
rushmorem 7b028de543 Fix mesos-0.22 build and update to 0.22.1 2015-05-14 18:02:09 +02:00
Pascal Wittmann 9c9c586534 Fixed several "package has no version" warnings 2015-05-14 17:19:41 +02:00
Charles Strahan 7493d6f1f6 marathon: 0.8.0 -> 0.8.1 2015-05-10 19:56:46 -04:00
Charles Strahan 9d9e21c7fd mesos: 0.21.0 -> 0.22.0
This also enables the network isolation feature that was introduced in
Mesos 0.20.0.
2015-05-10 17:23:20 -04:00
Sven Keidel 456da7fede apache pig: new package 2015-04-29 14:33:24 +02:00
Jaka Hudoklin f5bc58d733 Merge pull request #7545 from offlinehacker/pkgs/kubernetes/fixes
kubernetes: update to 0.15, module fixes
2015-04-28 14:12:53 +02:00
Pascal Wittmann f99d5cf2fc Fixed some descriptions 2015-04-28 10:55:20 +02:00
Jaka Hudoklin cbf1c3ca22 kubernetes: update to 0.15 2015-04-25 14:18:05 +02:00
Jaka Hudoklin 89f41b90fa Merge pull request #6874 from offlinehacker/pkgs/kubernetes/update
kubernetes: update to 0.12.1, fix module
2015-03-20 11:22:48 +01:00
Jaka Hudoklin 2482b75eae kubernetes: update to 0.12.1, fix module 2015-03-18 14:37:03 +01:00
Jaka Hudoklin 079520ced5 panamax: update, fix module 2015-03-18 14:35:24 +01:00
Jaka Hudoklin ebfadc2fc9 Merge pull request #6452 from rushmorem/update-chronos-port
Update chronos default port
2015-02-19 14:13:00 +01:00
rushmorem 74b40e9a43 Add marathon mesos framework 2015-02-19 13:30:00 +02:00
rushmorem a661802d29 Update chronos default port
Update chronos default port to match the one documented on
their website (http://airbnb.github.io/chronos). The one in
their repo (the current one) clashes with the marathon documented
one.
2015-02-19 12:31:12 +02:00
Matej Cotman 8967a2b32d panamax: rewrite using 'pleasent ruby' 2015-02-07 15:32:20 +01:00
Jaka Hudoklin ae34f923f7 panamax: simplify, refactor, add test 2015-02-07 15:32:20 +01:00
Matej Cotman 8c79a2df63 panamax: new package and service 2015-02-07 15:32:20 +01:00
John Wiegley 28b6fb61e6 Change occurrences of gcc to the more general cc
This is done for the sake of Yosemite, which does not have gcc, and yet
this change is also compatible with Linux.
2014-12-26 11:06:21 -06:00
Jaka Hudoklin 4a5897be65 spark: update to 1.1.1, simplify a lot 2014-12-04 17:40:21 +01:00
Jaka Hudoklin 36a5384ec9 mesos: update to 0.21.0 2014-12-02 12:22:12 +01:00
Jaka Hudoklin aeba2947b0 Add chronos, fault tolerant job scheduler for Mesos 2014-12-02 12:22:05 +01:00
Jaka Hudoklin eb2f9f029a kubernetes: build only on x86_64-linux 2014-11-30 01:24:59 +01:00