ARC Runners Not Being Created in EKS Cluster #3810

omerap12 · 2024-11-15T09:04:38Z

Checks

I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
I am using charts that are officially provided

Controller Version

ghcr.io/actions/gha-runner-scale-set-controller:0.9.3

Deployment Method

Helm

Checks

This isn't a question or user support case (For Q&A and community support, go to Discussions).
I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install the contoller using helm.

NAME	NAMESPACE  	REVISION	UPDATED                             	STATUS  	CHART                                	APP VERSION
arc 	arc-systems	1       	2024-07-02 12:48:26.029989 +0300 IDT	deployed	gha-runner-scale-set-controller-0.9.3	0.9.3

Install simple runner ( values file are attached ).

NAME                 	NAMESPACE  	REVISION	UPDATED                             	STATUS  	CHART                     	APP VERSION
arc-runner-set-mis   	arc-runners	1       	2024-11-15 10:04:35.538126 +0200 IST	deployed	gha-runner-scale-set-0.9.3	0.9.3

Describe the bug

We have installed arc runner set in our eks cluster:

 ~/ k get autoscalingrunnersets.actions.github.com -n arc-runners mis                                                                                                                                                           
NAME   MINIMUM RUNNERS   MAXIMUM RUNNERS   CURRENT RUNNERS   STATE   PENDING RUNNERS   RUNNING RUNNERS   FINISHED RUNNERS   DELETING RUNNERS
mis    4                 5                 5                         4

as you can see the CURRENT RUNNERS is 5.
However, when I check the pods, I get the following:

 ~/ k get pods -n arc-runners                                                                                                                                                                                                 
No resources found in arc-runners namespace.

This indicates that the minimum runner count isn't being respected, and runners aren't being created in response to GitHub hooks.

What's puzzling is that I have another runner set, using the same values file (with only a different githubConfigUrl and runnerScaleSetName), and it's working perfectly fine.

Describe the expected behavior

Runners should be created.

Additional Context

USER-SUPPLIED VALUES:
githubConfigSecret: pre-defined-secret-mis
githubConfigUrl: https://github.com/Company-MIS
maxRunners: 5
minRunners: 0
requests:
  cpu: 500m
  memory: 1Gi
runnerScaleSetName: mis
template:
  spec:
    containers:
    - command:
      - /home/runner/run.sh
      env:
      - name: DOCKER_HOST
        value: unix:///var/run/docker.sock
      image: company/gh-actions-runner:latest
      imagePullPolicy: IfNotPresent
      name: runner
      resources:
        requests:
          cpu: 1400m
          memory: 1Gi
      volumeMounts:
      - mountPath: /home/runner/_work
        name: work
      - mountPath: /var/run
        name: dind-sock
      - mountPath: /home/runner/.docker
        name: docker
    imagePullSecrets:
    - name: regsecret
    initContainers:
    - command:
      - cp
      - /home/runner/.docker2/config.json
      - /home/runner/.docker/config.json
      image: ghcr.io/actions/actions-runner:latest
      name: custom-init
      volumeMounts:
      - mountPath: /home/runner/.docker
        name: docker
      - mountPath: /home/runner/.docker2/config.json
        name: regsecret
        subPath: config.json
    - args:
      - dockerd
      - --host=unix:///var/run/docker.sock
      - --group=$(DOCKER_GROUP_GID)
      env:
      - name: DOCKER_GROUP_GID
        value: "123"
      image: docker:dind
      name: dind
      restartPolicy: OnFailure
      securityContext:
        privileged: true
      volumeMounts:
      - mountPath: /home/runner/_work
        name: work
      - mountPath: /var/run
        name: dind-sock
      - mountPath: /home/runner/externals
        name: dind-externals
      - mountPath: /root/.docker/config.json
        name: regsecret
        subPath: config.json
    - command:
      - cp
      - -r
      - -v
      - /home/runner/externals/.
      - /home/runner/tmpDir/
      image: ghcr.io/actions/actions-runner:latest
      name: init-dind-externals
      restartPolicy: Always
      volumeMounts:
      - mountPath: /home/runner/tmpDir
        name: dind-externals
    volumes:
    - emptyDir: {}
      name: work
    - emptyDir: {}
      name: docker
    - emptyDir: {}
      name: dind-sock
    - emptyDir: {}
      name: dind-externals
    - name: regsecret
      secret:
        items:
        - key: .dockerconfigjson
          path: config.json
        secretName: regsecret

Controller Logs

2024-11-15T08:00:45Z	INFO	EphemeralRunner	Cleaning up resources after after ephemeral runner termination	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-rxkjl","namespace":"arc-runners"}, "phase": "Failed"}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Cleaning up the runner pod	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-rxkjl","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Pod is deleted	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-rxkjl","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Cleaning up the runner jitconfig secret	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-rxkjl","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunnerSet	Ephemeral runner counts	{"version": "0.9.3", "ephemeralrunnerset": {"name":"mis-r4pfg","namespace":"arc-runners"}, "pending": 0, "running": 0, "finished": 0, "failed": 2, "deleting": 0}
2024-11-15T08:00:45Z	INFO	EphemeralRunnerSet	Scaling comparison	{"version": "0.9.3", "ephemeralrunnerset": {"name":"mis-r4pfg","namespace":"arc-runners"}, "current": 2, "desired": 0}
2024-11-15T08:00:45Z	INFO	EphemeralRunnerSet	Deleting ephemeral runners (scale down)	{"version": "0.9.3", "ephemeralrunnerset": {"name":"mis-r4pfg","namespace":"arc-runners"}, "count": 2}
2024-11-15T08:00:45Z	INFO	EphemeralRunnerSet	No pending or running ephemeral runners running at this time for scale down	{"version": "0.9.3", "ephemeralrunnerset": {"name":"mis-r4pfg","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Secret is deleted	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-rxkjl","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	EphemeralRunner has already finished. Stopping reconciliation and waiting for EphemeralRunnerSet to clean it up	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-rxkjl","namespace":"arc-runners"}, "phase": "Failed"}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Cleaning up resources after after ephemeral runner termination	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-tsw5z","namespace":"arc-runners"}, "phase": "Failed"}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Cleaning up the runner pod	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-tsw5z","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Pod is deleted	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-tsw5z","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Cleaning up the runner jitconfig secret	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-tsw5z","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	Secret is deleted	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-tsw5z","namespace":"arc-runners"}}
2024-11-15T08:00:45Z	INFO	EphemeralRunner	EphemeralRunner has already finished. Stopping reconciliation and waiting for EphemeralRunnerSet to clean it up	{"version": "0.9.3", "ephemeralrunner": {"name":"mis-r4pfg-runner-tsw5z","namespace":"arc-runners"}, "phase": "Failed"}
2024-11-15T08:00:46Z	INFO	AutoscalingRunnerSet	Find existing ephemeral runner set	{"version": "0.9.3", "autoscalingrunnerset": {"name":"mis","namespace":"arc-runners"}, "name": "mis-r4pfg", "specHash": "58997d4cf6"}
2024-11-15T08:02:25Z	INFO	AutoscalingListener	Listener pod is terminated	{"version": "0.9.3", "autoscalinglistener": {"name":"mis-754b578d-listener","namespace":"arc-systems"}, "namespace": "arc-systems", "name": "mis-754b578d-listener", "reason": "Error", "message": ""}
2024-11-15T08:02:26Z	INFO	AutoscalingListener	Listener pod is terminated	{"version": "0.9.3", "autoscalinglistener": {"name":"mis-754b578d-listener","namespace":"arc-systems"}, "namespace": "arc-systems", "name": "mis-754b578d-listener", "reason": "Error", "message": ""}
2024-11-15T08:02:26Z	INFO	AutoscalingListener	Listener pod is terminated	{"version": "0.9.3", "autoscalinglistener": {"name":"mis-754b578d-listener","namespace":"arc-systems"}, "namespace": "arc-systems", "name": "mis-754b578d-listener", "reason": "Error", "message": ""}
2024-11-15T08:02:26Z	INFO	AutoscalingListener	Creating a listener pod	{"version": "0.9.3", "autoscalinglistener": {"name":"mis-754b578d-listener","namespace":"arc-systems"}}
2024-11-15T08:19:26Z	INFO	EphemeralRunnerSet	Scaling comparison	{"version": "0.9.3", "ephemeralrunnerset": {"name":"mis-v7mzn","namespace":"arc-runners"}, "current": 5, "desired": 4}
2024-11-15T08:19:26Z	INFO	EphemeralRunnerSet	Deleting ephemeral runners (scale down)	{"version": "0.9.3", "ephemeralrunnerset": {"name":"mis-v7mzn","namespace":"arc-runners"}, "count": 1}
2024-11-15T08:19:26Z	INFO	EphemeralRunnerSet	No pending or running ephemeral runners running at this time for scale down	{"version": "0.9.3", "ephemeralrunnerset": {"name":"mis-v7mzn","namespace":"arc-runners"}}
2024-11-15T08:19:26Z	INFO	AutoscalingRunnerSet	Find existing ephemeral runner set	{"version": "0.9.3", "autoscalingrunnerset": {"name":"mis","namespace":"arc-runners"}, "name": "mis-v7mzn", "specHash": "58997d4cf6"}



### Runner Pod Logs

```shell
2024-11-15T08:08:22Z	INFO	listener-app	app initialized
2024-11-15T08:08:22Z	INFO	listener-app	Starting listener
2024-11-15T08:08:22Z	INFO	listener-app	refreshing token	{"githubConfigUrl": "https://github.com/Company-MIS"}
2024-11-15T08:08:22Z	INFO	listener-app	getting access token for GitHub App auth	{"accessTokenURL": "https://api.github.com/app/installations/52408526/access_tokens"}
2024-11-15T08:08:22Z	INFO	listener-app	getting runner registration token	{"registrationTokenURL": "https://api.github.com/orgs/Company-MIS/actions/runners/registration-token"}
2024-11-15T08:08:22Z	INFO	listener-app	getting Actions tenant URL and JWT	{"registrationURL": "https://api.github.com/actions/runner-registration"}
2024-11-15T08:08:22Z	INFO	listener-app.listener	Current runner scale set statistics.	{"statistics": "{\"totalAvailableJobs\":0,\"totalAcquiredJobs\":0,\"totalAssignedJobs\":1,\"totalRunningJobs\":0,\"totalRegisteredRunners\":0,\"totalBusyRunners\":0,\"totalIdleRunners\":0}"}
2024-11-15T08:08:22Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 1, "decision": 5, "min": 4, "max": 5, "currentRunnerCount": 5, "jobsCompleted": 0}
2024-11-15T08:08:22Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":5,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2024-11-15T08:08:22Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":5}}"}
2024-11-15T08:08:22Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "arc-runners", "name": "mis-v7mzn", "replicas": 5}
2024-11-15T08:08:22Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2024-11-15T08:28:36Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 4, "min": 4, "max": 5, "currentRunnerCount": 4, "jobsCompleted": 0}
2024-11-15T08:28:36Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":4,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2024-11-15T08:28:36Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":4}}"}
2024-11-15T08:28:36Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "arc-runners", "name": "mis-v7mzn", "replicas": 4}
2024-11-15T08:28:36Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 2}
2024-11-15T08:29:26Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 4, "min": 4, "max": 5, "currentRunnerCount": 4, "jobsCompleted": 0}
2024-11-15T08:29:26Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":4,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2024-11-15T08:29:26Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":4}}"}
2024-11-15T08:29:27Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "arc-runners", "name": "mis-v7mzn", "replicas": 4}
2024-11-15T08:29:27Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 2}

The text was updated successfully, but these errors were encountered:

omerap12 · 2024-11-15T09:34:11Z

How can I find the reason for the following status?

 ~/ k describe autoscalingrunnersets.actions.github.com -n arc-runners mis
Status:
  Current Runners:            5
  Failed Ephemeral Runners:   1
  Pending Ephemeral Runners:  4

omerap12 added bug Something isn't working gha-runner-scale-set Related to the gha-runner-scale-set mode needs triage Requires review from the maintainers labels Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARC Runners Not Being Created in EKS Cluster #3810

ARC Runners Not Being Created in EKS Cluster #3810

omerap12 commented Nov 15, 2024

omerap12 commented Nov 15, 2024

ARC Runners Not Being Created in EKS Cluster #3810

ARC Runners Not Being Created in EKS Cluster #3810

Comments

omerap12 commented Nov 15, 2024

Checks

Controller Version

Deployment Method

Checks

To Reproduce

Describe the bug

Describe the expected behavior

Additional Context

Controller Logs

omerap12 commented Nov 15, 2024