Affects Version/s: None
Fix Version/s: None
It would be great if Infusion PRs could be tested by CI. GPII's CI automation (and automated PR testing) hasn't been replicated for Fluid due to Jenkins limitations, namely:
- An unmaintained Jenkins PR builder plugin
- Buggy multijob plugin that doesn't honour timeout values
- A vast world of pain that is Jenkins plugins management
We've looked into GitLab CI which addresses the issues above and also provides nice features such as pipelines, simpler YAML job configuration files, pull-based CI agents, and a more pleasant UI. The main disadvantage with this option is that GitLab's CI offering is packaged as a larger offering meant to replace GitHub. They provide very limited GitHub repository mirroring which is triggered on an hourly basis. GitHub PRs can't be tested using just their software – more information is logged in the GPII-2473 issue.
Recently I came across Buildkite, a CI service that addresses all of the Jenkins and GitLab shortcomings. They provide free accounts to open source projects so I set up a test Fluid organization. You will need to create an account and be invited to the org to see CI results. Here is an example of an Infusion fork's PRs and branch merges being tested:
The config file used to test the above:
The disadvantages of using Buildkite are:
- Proprietary service which can't be self-hosted – although hosting Jenkins or GitLab involves considerable on-going maintenance costs in terms of securtity issues, testing new releases, support, etc.
- Job failures and successes are shown in the GitHub UI but Buildkite accounts are required to view detailed CI results
- I don't think PR jobs can be manually triggered using a confirmation string such as "ok to test" but I've contacted their support team to verify
- They don't provide a contributor whitelist feature so anyone can open a PR, potentially containing malicious code, and trigger CI jobs
There might be a workaround for the last point. When Michelle sent a test PR a
BUILDKITE_BUILD_CREATOR="michelled" environment variable was available to the Buildkite agent. A list of trusted GitHub account names could be maintained and checked against before running any jobs on the agent.