* runner: Expose dind runner dockerd logs via stdout/stderr
We've been letting supervisord to run dockerd within the dind runner container presuming it would avoid producing zombie processes. However we used dumb-init to wrap supervisord to wrap dockerd. In this picture supervisord might be unnecessary and dumb-init is actually a correct pid 0 for containers.
Rmoving supervisord removes this unnecessary complexity, while saving a little memory, and more importantly logs from dockerd is exposed via stdout/stderr of the container for easy access from kubectl-logs, fluentd, and so on.
* Add workflow job metrics to Github webhook server
* Fix handling of workflow_job.Conclusion
* Make the prometheus metrics exporter for the workflow jobs a dedicated application
* chart: Add support for deploying actions-metrics-server
* A few improvements to make it easy to cover in E2E
* chart: Add missing actionsmetrics.service.yaml
* chart: Do not modify actionsMetricsServer.replicaCount
* chart: Add documentation for actionsMetrics and actionsMetricsServer
Co-authored-by: Colin Heathman <[email protected]>
The runner graceful stop timeout has never been propagated to the dind sidecar due to configuration error in E2E. This fixes it, so that we can verify that the dind sidecar prestop can respect the graceful stop timeout.
Related to #1759
This fixes the said issue that I found while I was running a series of E2E tests to test other features and pull requestes I have recently contributed.
* ci: Fix runner builds but not pushes for forks
I noticed that our runners workflow is failing on docker-login due to that a pull request workflow job from a fork does not have access to repo secrets.
https://github.com/malachiobadeyi/actions-runner-controller/actions/runs/3390463793/jobs/5634638183
Can we try this, so that hopefully it suppresses docker-login for pull requests from forks?
* Update .github/workflows/runners.yaml
* fixup! Update .github/workflows/runners.yaml
Signed-off-by: Yusuke Kuoka <[email protected]>
* fixup! fixup! Update .github/workflows/runners.yaml
Signed-off-by: Yusuke Kuoka <[email protected]>
Signed-off-by: Yusuke Kuoka <[email protected]>
* Fix permission issue when you use PV for rootless dind cache
This fixes the said issue I have found while testing #1759.
Signed-off-by: Yusuke Kuoka <[email protected]>
While testing #1759, I found an issue in the rootless dind entrypoint that it was not respecting the configured MTU for dind docker due to a permission issue. This fixes that.
It turned out too hard to debug configuration issues on the rootless dind daemon as it was not writing any logs to stdout/stderr of the container. This fixes that, so that any rootless dind configuration or startup errors are visible in e.g. the kubectl-logs output.
* Let it be a bug only when it's reproducible with official runner image
A custom runner image can break runners and ARC in interesting ways. Probably it's better to clearly state that ARC is not guaranteed to work with every custom runner image in the wild.
* Update .github/ISSUE_TEMPLATE/bug_report.yml
* Add workflow for validating runner scripts with shellcheck
I am about to revisit #1517, #1454, #1561, and #1560 as a part of our on-going effort to a major enhancement to the runner entrypoints being made in #1759.
This change is a counterpart of #1852. #1852 enables you to easily run shellcheck locally. This enables you to automatically run shellcheck on every pull request.
Currently, any shellcheck error does not result in failing the workflow job. Once we addressed all the shellcheck findings, we can flip the fail_on_error option to true and let jobs start failing on pull requests that introduce invalid or suspicious bash code.
so that ARC respect the registration timeout, terminationGracePeriodSeconds and RUNNER_GRACEFUL_STOP_TIMEOUT(#1759) when the runner pod was terminated externally too early after its creation
While I was running E2E tests for #1759, I discovered a potential issue that ARC can terminate runner pods without waiting for the registration timeout of 10 minutes.
You won't be affected by this in normal circumstances, as this failure scenario can be triggered only when you or another K8s controller like cluster-autoscaler deleted the runner or the runner pod immediately after the runner or the runner pod has been created. But probably is it worth fixing it anyway because it's not impossible to trigger it?
I am about to revisit #1517, #1454, #1561, and #1560 as a part of our on-going effort for a major enhancement to the runner entrypoints being made in #1759.
This commit adds the makefile target to run shellformat locally, so that any contributor can use it before submitting a pull request.
I am about to revisit #1517, #1454, #1561, and #1560 as a part of our on-going effort for a major enhancement to the runner entrypoints being made in #1759.
This change updates and reintroduces #1517 contributed by @CASABECI in a way it becomes applicable to today's code-base.
Both fields can be useless when the reporter thought only one or two lines of the respective logs are relevant and it turned out we had to see another line later.
To avoid such a situation I'd like to change the field labels to include `Whole` so that it looks like `Whole Runner Pod Logs`, and ask to not omit the logs in the field description.
Honestly, I'm a bit tired of seeing issues filed with the default title "Bug" that are seemingly not related to real bugs!
I'm emptying it so that the reporter is more encourage to write up a sentence to describe the problem and they hopefully notice along the way that there are places to diagnose before considering it as a bug.
I think we should at least recommend the reporter read the troubleshooting guide before submitting a bug report. It would give them more chances to realize it isn't a bug in some cases.
Even though we have a bug report form, we've seen so many issues being written in freestyle and I think every occurrence of it resulted in needing multiple hops for us to gather the information necessary for diagnosing the issue. I hope every bug report is more actionable. Now we disable blank issues so that the reporter is likely to choose the bug report form or Discussions depending on their situation.
* chart: Remove support for extensions/v1beta1 and networking.k8s.io/v1beta1
`networking.k8s.io/v1` has been available since v1.19.
As of today, AWS EKS supports v1.19+ and Oracle Cloud supports v1.20+. GKE and AKS supports v1.21+. The upstream Kubernetes project maintains v1.22+.
So it should be safe to remove it now.
* fixup! chart: Remove support for extensions/v1beta1 and networking.k8s.io/v1beta1
This removes the flag and code for the legacy GitHub API cache. We already migrated to fully use the new HTTP cache based API cache functionality which had been added via #1127 and available since ARC 0.22.0. Since then, the legacy one had been no-op and therefore removing it is safe.
Ref #1412
This contains apparently enough changes to the current E2E test code to make it runnable against remote Kubernetes clusters. I was actually able to make the test passing against my AWS EKS based test clusters with these changes. You still need to trigger it manually from a local checkout of the ARC repo today. But this might be the foundation for automated E2E tests against major cloud providers.
I encountered this once while E2E testing ARC with K8s 1.22 and cert-manager 1.1.1. The K8s version is too high / The cert-manager is too low so you generally need to fix either. In a standard scenario, it should be more feasible and meaningful to upgrade cert-manager to a recent enough version that supports the new Kubernetes version.
I've been manually setting up Argo Tunnel to expose the webhook server while running E2E tests so that I can cover the webhook-based autoscaling. This automates the setup process so that we can automatiaclly bring up and down cloudflared before/after the test run, so that it can be a part of our upcoming automated E2E test.
The regression resulted in the webhook-based autoscaler be unable to find visible runner groups and therefore unable to scale up and down the target RunnerDeployment/RunnerSet at all when the webhook-based autoscaler was provided GitHub API credentials to enable the runner groups support. This fixes that.
The regression was introduced via #1578 which is not released yet. Users of existing ARC releases are therefore not affected.
* Cover ARC upgrade in E2E test
so that we can make it extra sure that you can upgrade the existing installation of ARC to the next and also (hopefully) it is backward-compatible, or at least it does not break immediately after upgrading.
* Consolidate E2E tests for RS and RD
* Fix E2E for RD to pass
* Add some comment in E2E for how to release disk consumed after dozens of test runs
The primary goal of this change is to let the tester know about the config difference between the explicitly configured ephemeral work volume vs the automatically configured work volume with workVolumeClaimTemplate+containerMode=kubernetes.
