Starting ARC v0.27.2, we've changed the `docker.sock` path from `/var/run/docker.sock` to `/var/run/docker/docker.sock`. That resulted in breaking some container-based actions due to the hard-coded `docker.sock` path in various places.
Even `actions/runner` seem to use `/var/run/docker.sock` for building container-based actions and for service containers?
Anyway, this fixes that by moving the sock file back to the previous location.
Once this gets merged, users stuck at ARC v0.27.1, previously upgraded to 0.27.2 or 0.27.3 and reverted back to v0.27.1 due to #2519, should be able to upgrade to the upcoming v0.27.4.
Resolves#2519Resolves#2538
#2490 has been happening since v0.27.2 for non-dind runners based on Ubuntu 20.04 runner images. It does not affect Ubuntu 22.04 non-dind runners(i.e. runners with dockerd sidecars) and Ubuntu 20.04/22.04 dind runners(i.e. runners without dockerd sidecars). However, presuming many folks are still using Ubuntu 20.04 runners and non-dind runners, we should fix it.
This change tries to fix it by defaulting to the docker group id 1001 used by Ubuntu 20.04 runners, and use gid 121 for Ubuntu 22.04 runners. We use the image tag to see which Ubuntu version the runner is based on. The algorithm is so simple- we assume it's Ubuntu-22.04-based if the image tag contains "22.04".
This might be a breaking change for folks who have already upgraded to Ubuntu 22.04 runners using their own custom runner images. Note again; we rely on the image tag to detect Ubuntu 22.04 runner images and use the proper docker gid- Folks using our official Ubuntu 22.04 runner images are not affected. It is a breaking change anyway, so I have added a remedy-
ARC got a new flag, `--docker-gid`, which defaults to `1001` but can be set to `121` or whatever gid the operator/admin likes. This can be set to `--docker-gid=121`, for example, if you are using your own custom runner image based on Ubuntu 22.04 and the image tag does not contain "22.04".
Fixes#2490
* runner: Expose dind runner dockerd logs via stdout/stderr
We've been letting supervisord to run dockerd within the dind runner container presuming it would avoid producing zombie processes. However we used dumb-init to wrap supervisord to wrap dockerd. In this picture supervisord might be unnecessary and dumb-init is actually a correct pid 0 for containers.
Rmoving supervisord removes this unnecessary complexity, while saving a little memory, and more importantly logs from dockerd is exposed via stdout/stderr of the container for easy access from kubectl-logs, fluentd, and so on.
* Add workflow job metrics to Github webhook server
* Fix handling of workflow_job.Conclusion
* Make the prometheus metrics exporter for the workflow jobs a dedicated application
* chart: Add support for deploying actions-metrics-server
* A few improvements to make it easy to cover in E2E
* chart: Add missing actionsmetrics.service.yaml
* chart: Do not modify actionsMetricsServer.replicaCount
* chart: Add documentation for actionsMetrics and actionsMetricsServer
Co-authored-by: Colin Heathman <[email protected]>
The runner graceful stop timeout has never been propagated to the dind sidecar due to configuration error in E2E. This fixes it, so that we can verify that the dind sidecar prestop can respect the graceful stop timeout.
Related to #1759
This fixes the said issue that I found while I was running a series of E2E tests to test other features and pull requestes I have recently contributed.
* ci: Fix runner builds but not pushes for forks
I noticed that our runners workflow is failing on docker-login due to that a pull request workflow job from a fork does not have access to repo secrets.
https://github.com/malachiobadeyi/actions-runner-controller/actions/runs/3390463793/jobs/5634638183
Can we try this, so that hopefully it suppresses docker-login for pull requests from forks?
* Update .github/workflows/runners.yaml
* fixup! Update .github/workflows/runners.yaml
Signed-off-by: Yusuke Kuoka <[email protected]>
* fixup! fixup! Update .github/workflows/runners.yaml
Signed-off-by: Yusuke Kuoka <[email protected]>
Signed-off-by: Yusuke Kuoka <[email protected]>
* Fix permission issue when you use PV for rootless dind cache
This fixes the said issue I have found while testing #1759.
Signed-off-by: Yusuke Kuoka <[email protected]>
While testing #1759, I found an issue in the rootless dind entrypoint that it was not respecting the configured MTU for dind docker due to a permission issue. This fixes that.
It turned out too hard to debug configuration issues on the rootless dind daemon as it was not writing any logs to stdout/stderr of the container. This fixes that, so that any rootless dind configuration or startup errors are visible in e.g. the kubectl-logs output.
* Let it be a bug only when it's reproducible with official runner image
A custom runner image can break runners and ARC in interesting ways. Probably it's better to clearly state that ARC is not guaranteed to work with every custom runner image in the wild.
* Update .github/ISSUE_TEMPLATE/bug_report.yml
* Add workflow for validating runner scripts with shellcheck
I am about to revisit #1517, #1454, #1561, and #1560 as a part of our on-going effort to a major enhancement to the runner entrypoints being made in #1759.
This change is a counterpart of #1852. #1852 enables you to easily run shellcheck locally. This enables you to automatically run shellcheck on every pull request.
Currently, any shellcheck error does not result in failing the workflow job. Once we addressed all the shellcheck findings, we can flip the fail_on_error option to true and let jobs start failing on pull requests that introduce invalid or suspicious bash code.
so that ARC respect the registration timeout, terminationGracePeriodSeconds and RUNNER_GRACEFUL_STOP_TIMEOUT(#1759) when the runner pod was terminated externally too early after its creation
While I was running E2E tests for #1759, I discovered a potential issue that ARC can terminate runner pods without waiting for the registration timeout of 10 minutes.
You won't be affected by this in normal circumstances, as this failure scenario can be triggered only when you or another K8s controller like cluster-autoscaler deleted the runner or the runner pod immediately after the runner or the runner pod has been created. But probably is it worth fixing it anyway because it's not impossible to trigger it?
I am about to revisit #1517, #1454, #1561, and #1560 as a part of our on-going effort for a major enhancement to the runner entrypoints being made in #1759.
This commit adds the makefile target to run shellformat locally, so that any contributor can use it before submitting a pull request.
I am about to revisit #1517, #1454, #1561, and #1560 as a part of our on-going effort for a major enhancement to the runner entrypoints being made in #1759.
This change updates and reintroduces #1517 contributed by @CASABECI in a way it becomes applicable to today's code-base.
Both fields can be useless when the reporter thought only one or two lines of the respective logs are relevant and it turned out we had to see another line later.
To avoid such a situation I'd like to change the field labels to include `Whole` so that it looks like `Whole Runner Pod Logs`, and ask to not omit the logs in the field description.
Honestly, I'm a bit tired of seeing issues filed with the default title "Bug" that are seemingly not related to real bugs!
I'm emptying it so that the reporter is more encourage to write up a sentence to describe the problem and they hopefully notice along the way that there are places to diagnose before considering it as a bug.
I think we should at least recommend the reporter read the troubleshooting guide before submitting a bug report. It would give them more chances to realize it isn't a bug in some cases.
Even though we have a bug report form, we've seen so many issues being written in freestyle and I think every occurrence of it resulted in needing multiple hops for us to gather the information necessary for diagnosing the issue. I hope every bug report is more actionable. Now we disable blank issues so that the reporter is likely to choose the bug report form or Discussions depending on their situation.
* chart: Remove support for extensions/v1beta1 and networking.k8s.io/v1beta1
`networking.k8s.io/v1` has been available since v1.19.
As of today, AWS EKS supports v1.19+ and Oracle Cloud supports v1.20+. GKE and AKS supports v1.21+. The upstream Kubernetes project maintains v1.22+.
So it should be safe to remove it now.
* fixup! chart: Remove support for extensions/v1beta1 and networking.k8s.io/v1beta1
This removes the flag and code for the legacy GitHub API cache. We already migrated to fully use the new HTTP cache based API cache functionality which had been added via #1127 and available since ARC 0.22.0. Since then, the legacy one had been no-op and therefore removing it is safe.
Ref #1412
This contains apparently enough changes to the current E2E test code to make it runnable against remote Kubernetes clusters. I was actually able to make the test passing against my AWS EKS based test clusters with these changes. You still need to trigger it manually from a local checkout of the ARC repo today. But this might be the foundation for automated E2E tests against major cloud providers.
I encountered this once while E2E testing ARC with K8s 1.22 and cert-manager 1.1.1. The K8s version is too high / The cert-manager is too low so you generally need to fix either. In a standard scenario, it should be more feasible and meaningful to upgrade cert-manager to a recent enough version that supports the new Kubernetes version.
I've been manually setting up Argo Tunnel to expose the webhook server while running E2E tests so that I can cover the webhook-based autoscaling. This automates the setup process so that we can automatiaclly bring up and down cloudflared before/after the test run, so that it can be a part of our upcoming automated E2E test.
