Commit graph

137 commits

Author SHA1 Message Date
Frederik Rietdijk f8c6921e97
hydra: wrap executables with hydra env vars
We already set the relevant env vars in the systemd services. That does
not help one when executing any of the executables outside a service,
e.g. when creating a new user.
2020-04-11 14:36:42 +02:00
Maximilian Bosch 0f5c38feed
hydra: 2020-03-24 -> 2020-04-07
Also removed `pkgs.hydra-flakes` since flake-support has been merged
into master[1]. Because of that, `pkgs.hydra-unstable` is now compiled
against `pkgs.nixFlakes` and currently requires a patch since Hydra's
master doesn't compile[2] atm.

[1] https://github.com/NixOS/hydra/pull/730
[2] https://github.com/NixOS/hydra/pull/732
2020-04-07 14:11:12 +02:00
Jörg Thalheim 212b574d89
Merge pull request #81298 from Mic92/buildkite
nixos/buildkite-agents: don't run as nogroup
2020-04-02 13:39:34 +01:00
Florian Klink 9faea55282
Merge pull request #74379 from bachp/gitlab-runner-reload
nixos/gitlab-runner: reload on config change
2020-03-31 20:18:28 +02:00
Maximilian Bosch bd5324c4fc
hydra: 2020-02-06 -> 2020-03-{24,27}
Upgrades Hydra to the latest master/flake branch. To perform this
upgrade, it's needed to do a non-trivial db-migration which provides a
massive performance-improvement[1].

The basic ideas behind multi-step upgrades of services between NixOS versions
have been gathered already[2]. For further context it's recommended to
read this first.

Basically, the following steps are needed:

* Upgrade to a non-breaking version of Hydra with the db-changes
  (columns are still nullable here). If `system.stateVersion` is set to
  something older than 20.03, the package will be selected
  automatically, otherwise `pkgs.hydra-migration` needs to be used.

* Run `hydra-backfill-ids` on the server.

* Deploy either `pkgs.hydra-unstable` (for Hydra master) or
  `pkgs.hydra-flakes` (for flakes-support) to activate the optimization.

The steps are also documented in the release-notes and in the module
using `warnings`.

`pkgs.hydra` has been removed as latest Hydra doesn't compile with
`pkgs.nixStable` and to ensure a graceful migration using the newly
introduced packages.

To verify the approach, a simple vm-test has been added which verifies
the migration steps.

[1] https://github.com/NixOS/hydra/pull/711
[2] https://github.com/NixOS/nixpkgs/pull/82353#issuecomment-598269471
2020-03-28 23:33:25 +01:00
Jörg Thalheim 8b7f4fa8a6
nixos/buildkite-agents: don't run as nogroup 2020-02-28 15:34:37 +00:00
Yorick van Pelt 1b351f81f4
nixos/buildkite-agents: fix hooksDir assertion 2020-02-19 12:22:35 +01:00
Yorick van Pelt e242eccb0b
nixos/buildkite-agents: update release notes 2020-02-10 13:36:30 +01:00
Yorick van Pelt f003810989
nixos/buildkite-agents: support multiple buildkite agents 2020-02-10 13:35:14 +01:00
Florian Klink 0daae2e08c nixos/buildkite: drop user option (#78160)
* nixos/buildkite: drop user option

This reverts 8c6b1c3eaa.

Turns out, buildkite-agent has logic to write .ssh/known_hosts files and
only really works when $HOME and the user homedir are in sync.

On top of that, we provision ssh keys in /var/lib/buildkite-agent, which
doesn't work if that other users' homedir points elsewhere (we can cheat
by setting $HOME, but then getent and $HOME provide conflicting
results).

So after all, it's better to only run the system-wide buildkite agent as
the "buildkite-agent" user only - if one wants to run buildkite as
different users, systemd user services might be a better fit.

* nixosTests.buildkite-agent: add node with separate user and no ssh key
2020-01-21 13:21:57 +00:00
Florian Klink 4b73d3c444 nixos/buildkite: make privateSshKeyPath optional
When only cloning public repos, or when the ssh key is provided by
different means, we don't need to manage it here.
2020-01-20 11:10:56 +01:00
Florian Klink 70308a7daf nixos/buildkite-agent: add gnutar, gzip and git to runtimePackages
These are required for nix to do builtins.fetchTarball and
builtins.fetchGit, so most likely we want them to be around.
2020-01-20 10:30:24 +01:00
Florian Klink 7838f00824 nixos/buildkite: stop using deprecated option 2020-01-20 10:30:22 +01:00
Florian Klink 8c6b1c3eaa nixos/buildkite-agent: add "user" option
This allows buildkite-agent to run as another user.

It'll still run builds from /var/lib/buildkite-agent and setup things in
there.
2020-01-19 21:21:15 +01:00
Florian Klink 63c310c07e nixos/buildkite-agent: change option meta-data into tags attrset
This is called tags in the buildkite 3.
2020-01-17 23:11:26 +01:00
Yorick van Pelt e1ad9d1685 nixos/buildkite-agent: add a shell attribute to specify the shell being used
This gets passed to BUILDKITE_SHELL, which will specify the shell being
used to executes script in.

Defaults to `${pkgs.bash}/bin/bash -e -c`, matching how buildkite
behaves on other distros.
2020-01-17 18:40:11 +01:00
Florian Klink 99d0f4c380 nixos/buildkite-agent: tweak TimeoutStopSec and KillMode
This improves behaviour when the service is being stopped.
2020-01-17 18:40:11 +01:00
Florian Klink 9237767277 nixos/buildkite-agent: actually use services.buildkite-agent.package
We were currently just using pkgs.buildkite-agent, no matter what was
configured in services.buildkite-agent.package
2020-01-17 18:40:11 +01:00
Florian Klink c154dd5101 nixos/buildkite-agent: stop deploying public keys, make ssh private key optional.
SSH public keys aren't needed to clone private repos, and if we only
need to configure a single attribute, there's no need for the "openssh"
attrset anymore.
2020-01-17 18:40:11 +01:00
Mark Karpov a8cdfd30ff
Make buildkite agent service compatible with buildkite-agent3 2020-01-15 17:16:44 +01:00
Bas van Dijk b0b37569a7
Merge pull request #77669 from basvandijk/hydra-sync-buildMachinesFiles-with-upstream
hydra: only set buildMachinesFiles when nix.buildMachines is defined
2020-01-15 09:39:25 +01:00
Bas van Dijk ad554b3e4e hydra: only set buildMachinesFiles when nix.buildMachines is defined
This applies [hydra PR #432](https://github.com/NixOS/hydra/pull/432)
to the NixOS module in nixpkgs:

```
commit 4efd078977e5ea20e1104783efc324cba11690bc
Author: Bas van Dijk <v.dijk.bas@gmail.com>
Date:   Sun Dec 11 15:35:38 2016 +0100

    Only set buildMachinesFiles when nix.buildMachines is defined
```
2020-01-14 12:32:40 +01:00
Bas van Dijk f34aaa6646 hydra: remove the build-use-substitutes option
The following commit from 2016 in hydra removed the `--option
build-use-substitutes` from the hydra-queue-runner service:

```
commit ee2e9f5335c8c0288c102975b506f6b275793cfe
Author: Eelco Dolstra <edolstra@gmail.com>
Date:	Fri Oct 7 20:23:05 2016 +0200

    Update to reflect BinaryCacheStore changes

    BinaryCacheStore no longer implements buildPaths() and ensurePath(),
    so we need to use copyPath() / copyClosure().
```

It would be better if the hydra module in NixOS matches the upstream
module.
2020-01-14 12:14:25 +01:00
worldofpeace 2af959e579 nixos/jenkins/slave: fix eval 2020-01-12 15:17:05 -05:00
worldofpeace 79de730e77 nixos/buildbot: fix eval
Fixes https://github.com/NixOS/nixpkgs/issues/77309
2020-01-08 10:06:08 -05:00
rnhmjoj 1d61efb7f1 treewide: use attrs instead of list for types.loaOf options 2020-01-06 10:39:18 -05:00
Ben Gamari 848af2c6fc nixos/gitlab-runner: reload on config change
With this change it is no longer required to restart the runner on every
change. Instead it can just reload it's config while running.
2019-11-27 23:00:29 +01:00
Silvan Mosberger dd0a47e7ae
treewide: Switch to system users (#71055)
treewide: Switch to system users
2019-11-01 13:26:43 +01:00
edef a68219b79b nixos/nix-daemon: don't refer to deprecated option aliases
The `gc-` prefixed versions of these options have been
deprecated since Nix 2.0.
2019-10-18 12:27:43 +00:00
Janne Heß d6c08776ba treewide: Switch to system users 2019-10-12 22:25:28 +02:00
Alex Whitt f01224374d nixos/jenkins: Copy plugins as .jpi to fix initialization errors 2019-09-17 19:54:57 +02:00
Maximilian Bosch ce37a040c2
nixos/hydra: incorporate upstream changes and update test
During the last update, `hydra-notify` was rewritten as a daemon which
listens to postgresql notifications for each build[1]. The module
uses the `hydra-notify.service` unit from upstream's Hydra module and
the VM test ensures that email notifications are sent properly.

Also updated `hydra-init.service` to install `pg_trgm` on a local
database if needed[2].

[1] c7861b85c4
[2] 8a0a5ec3a3
2019-09-14 12:58:42 +02:00
Frederik Rietdijk ad1d58c622 Merge staging-next into staging 2019-08-31 10:04:20 +02:00
Pascal Bach 02ed974bba nixos/gitlab-runner: add missing HOME to environment (#67450)
Gitlab runner fails to start if HOME is not set.
2019-08-28 20:27:28 +00:00
volth 08f68313a4 treewide: remove redundant rec 2019-08-28 11:07:32 +00:00
Daniel Schaefer 786f02f7a4 treewide: Remove usage of isNull
isNull "is deprecated; just write e == null instead" says the Nix manual
2019-04-29 14:05:50 +02:00
Ben Wolsieffer b2e11e0cdf buildbot: 1.8.1 -> 2.1.0 2019-03-22 18:43:15 -04:00
Symphorien Gibol a915b33315 nixos: add preferLocalBuild=true; on derivations for config files 2019-02-22 20:11:27 +01:00
Bjørn Forsman bb94d419fb nixos/jenkins-job-builder: add accessTokenFile option
The new option allows storing the secret access token outside the world
readable Nix store.
2018-12-03 17:07:29 +01:00
Bjørn Forsman 8ebfd5c45c nixos/jenkins-job-builder: stop reloadScript on error
Currently there are two calls to curl in the reloadScript, neither which
check for errors. If something is misconfigured (like wrong authToken),
the only trace that something wrong happened is this log message:

  Asking Jenkins to reload config
  <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='<'</pre>

The service isn't marked as failed, so it's easy to miss.

Fix it by passing --fail to curl.

While at it:
* Add $curl_opts and $jenkins_url variables to keep the curl command
  lines DRY.
* Add --show-error to curl to show short error message explanation when
  things go wrong (like HTTP 401 error).
* Lower-case the $CRUMB variable as upper case is for exported environment
  variables.

The new behaviour, when having wrong accessToken:

  Asking Jenkins to reload config
  curl: (22) The requested URL returned error: 401

And the service is clearly marked as failed in `systemctl --failed`.
2018-12-03 17:07:29 +01:00
Ben Wolsieffer 73c523a605 buildbot: add Python 3 support 2018-10-11 21:39:11 -04:00
volth 2e979e8ceb [bot] nixos/*: remove unused arguments in lambdas 2018-07-20 20:56:59 +00:00
Florian Klink fff5923686 nixos/modules: users.(extraUsers|extraGroup->users|group) 2018-06-30 03:02:58 +02:00
Rodney Lorrimar 3e446ecd56 nixos/buildkite-agent: fix variable expansion in hook scripts
@cleverca found this bug in the declarative hooks config. Any shell
variables referenced in a hook script would get expanded by the hooks
directory builder.

Prevent variable expansion by quoting the here doc limit string.
2018-04-27 12:17:40 +01:00
Corey O'Connor 28a55f5bd6 jenkins: Add "java.awt.headless=true" to JDK options.
This is consistent with the recommended jenkins startup script. See:

* https://wiki.jenkins.io/display/JENKINS/JenkinsLinuxStartupScript
2018-03-17 22:16:59 -07:00
Corey O'Connor 241160aacc jenkins: Include Deja-Vu fonts in system environment to silence jenkins warning. 2018-03-17 22:16:26 -07:00
Tuomas Tynkkynen 1e107443ad Merge commit '618ac29687a650d854c8bea7efd4490387589ce2' into staging 2018-03-03 22:40:32 +02:00
Bjørn Forsman 5de8f99f03 nixos/jenkins: ensure postStart follows redirects
Pass the -L flag to curl to make it follow redirects. This fixes an
issue I found when setting up reverse proxy for Jenkins. Without this
fix, the returned HTTP code was stuck at 302, making postStart fail the
service (it expects 200 or 403).
2018-03-02 08:48:38 +01:00
Shea Levy fec543436d
nixos: Move uses of stdenv.shell to runtimeShell. 2018-03-01 14:38:53 -05:00
Rodney Lorrimar e552633c20
nixos/buildkite-agent: add coreutils to PATH of service
This simplifies the service script and it's probable that many builds
will need coreutils anyway.
2018-02-20 08:17:14 +00:00