Operations
Troubleshooting
Section titled “Troubleshooting”The controller reports problems as events on the Runner, so start with:
kubectl -n <namespace> describe runner <name>kubectl -n <namespace> get runner,pod,pvckubectl -n <namespace> describe pod <pod>kubectl -n runner-controller-system logs deploy/runner-controller-controller-managerThe container of the runner in a runner pod is named runner: kubectl -n <namespace> logs <pod> -c runner shows what the runner does.
While a Runner keeps failing, the controller retries it with growing delays. Editing the Runner triggers an immediate retry.
Warning events
Section titled “Warning events”| Event | Cause and fix |
|---|---|
RuntimeNotSet |
The Runner sets neither runtime nor kubernetesRuntime, which only happens to Runner resources created before upgrading to 0.7.10. The controller stops acting on it. Add one of them, with its labels. |
InspectGiteaFailed with status 403 |
The license of Gitea Enterprise is missing or invalid. The controller creates no runner pods until it is fixed. |
InspectGiteaFailed with status 401 |
The registration token was reset or deleted in Gitea. Update the Secret of registrationTokenSecretRef (or registrationToken) with the current token. |
InspectGiteaFailed with status 404 |
instanceUrl does not point to a valid Gitea Enterprise instance, see Gitea URL. |
InspectGiteaFailed without a status |
The controller cannot reach Gitea: check instanceUrl, the network between the controller and Gitea, and the certificate, see Self-signed certificates. |
RegistrationTokenUnavailable |
The Secret of registrationTokenSecretRef, or its key, is missing or empty, or the value has leading or trailing whitespace, see Registration token. |
RegisterRunnerFailed |
The controller cannot register a runner with Gitea. Check instanceUrl, the token, and the network between the controller and Gitea. |
BuildRunnerWorkerFailed |
The Runner refers to something that is missing or invalid, named in the message: a Secret or ConfigMap of runnerEnv or runnerEnvFrom (runner environment), the S3 keys (cache S3 credentials), the external cache key (cache server external secret), a volumeName that is used twice or by the controller (is taken, is the controller's), an init container (runnerInitContainers), or a runner.file set in your own configuration, which should be left unset. Until it is fixed, the controller stops adjusting the runner pods of this Runner: it does not scale or apply changes, and the runner pods already running keep running. It is only checked while runner pods are needed. |
ApplyRunnerWorkerFailed with violates PodSecurity |
The namespace does not allow the runner pods: privileged ones with the Docker-in-Docker backend, or init containers that do not meet restricted with the Kubernetes backend. |
ApplyRunnerWorkerFailed with exists but is not managed by this Runner |
Kubernetes backend: a ServiceAccount or RoleBinding of the same name exists, see Permissions. |
InvalidJobPodTemplate |
Kubernetes backend: the job pod template is invalid, and the controller stops acting on the Runner until it is fixed, see Invalid templates. |
RunnerInitContainerFailed |
An init container failed or cannot start. The event has the exit code and the end of its log, kubectl -n <namespace> logs <pod> -c <init container> shows the whole log. |
RunnerRegisteredItself |
The runner did not use the registration the controller made, or another runner of the same name is online in the same scope. Check the log of the runner container. |
RunnerNotOnline |
The runner has run for 3 minutes without coming online in Gitea. Check the log of the runner container. If it tries to register, for example with token is empty, the image version does not work with the controller. |
DestroyWorker |
A runner pod did not come online and was created again with empty caches, see below. |
GracePeriodTooSmall |
terminationGracePeriodSeconds is below shutdownTimeout, so jobs can be terminated before they finish. |
JobTmpfsMemoryTooSmall |
The memory request or limit cannot hold jobTmpfs, see tmpfs memory. |
CachePVCClassChangeIgnored, CachePVCShrinkIgnored, CachePVCExpandUnsupported |
A storage change that needs a new volume, which allowDeleteCachePVC does not allow, see Changing StorageClass or size. |
Other symptoms
Section titled “Other symptoms”| Symptom | Cause and fix |
|---|---|
Creating or changing a Runner fails with one of runtime and kubernetesRuntime must be set or labels: Required value |
Set one of runtime and kubernetesRuntime, with the labels your workflows use in runs-on, see Labels. |
Volumes stay Pending |
No StorageClass of that name, or it does not provision dynamically. className is standard by default, see Volumes of a runner pod. kubectl describe pvc shows why. |
Pods Pending with volume node affinity conflict |
No node can run the pod where its volume is, see Storage requirements. If the volumes are bound to a node that no longer exists, delete them and the pod: kubectl -n <namespace> delete pvc <runner pod>-action-cache <runner pod>-cache --wait=false (-cache with Docker-in-Docker only), then kubectl -n <namespace> delete pod <runner pod>. The controller creates the pod again with new, empty volumes. |
Pods Pending with no matching node |
nodeSelector matches no node, a taint is not tolerated, or the cluster is full. |
| Scaling up takes minutes | This is expected when scaling up from zero: a cold start includes a new node, if needed, and the image pull. Use minSize to keep runners available. |
Runner pods are not rebuilt after a change, with RollDeferred events |
They run jobs, and are rebuilt once the jobs are done, see How configuration changes are applied. |
| Jobs fail to connect to Docker | Docker-in-Docker backend: the runner image is not a -dind variant. |
| With a proxy, the runners do not come online, the workflow cache stops working, or job pods cannot be created | NO_PROXY misses an address, see Proxy. |
| The workflow cache always misses with S3 | The log of the runner container shows errors with sync S3 cache or upload cache to S3: check the endpoint, region, keys and permissions, including the bucket policy. |
| The workflow cache always misses with the built-in cache server | The cluster domain is not cluster.local: set cacheServer.host, see Built-in cache server. |
forbidden errors for system:serviceaccount:runner-controller-system:runner-controller-controller-manager, in the controller log or in warning events |
The controller lacks a permission, see Missing permissions. |
Kubernetes backend: jobs fail with pods is forbidden |
The ServiceAccount of the runner lacks permissions, see Permissions. |
Kubernetes backend: jobs fail with violates PodSecurity |
The job pods do not meet the level of the namespace, see Pod Security level. |
Kubernetes backend: jobs fail to download actions, with permission denied in the log of the runner container |
The StorageClass does not support fsGroup, see Storage. With your own configuration, host.workdir_parent is missing, see Using your own runner configuration. |
Kubernetes backend: job pods are evicted for disk pressure or ephemeral-storage |
The node disk cannot hold the job images and workspaces, see Storage. |
Kubernetes backend: jobs wait with Waiting for pod ... to be scheduled |
No node fits the resources of the job container or the scheduling of the job pod template. |
| Kubernetes backend: the workflow cache times out | The job pods cannot reach port 8088 of their runner pod, check the NetworkPolicies. |
Runner pods that do not come online
Section titled “Runner pods that do not come online”When the runner of a runner pod has not been online in Gitea for 5 minutes, and the pod started at least 15 minutes ago, image pulls and init containers included, the controller deletes the pod with its configuration and its cache volumes, reports a DestroyWorker warning, and creates it again. The registration is kept, unless Gitea no longer lists that runner, in which case it is registered anew. Pods that have not started, such as Pending ones, are left alone.
This also applies to a runner that came online and went offline later. Before that, the controller reports RunnerNotOnline after 3 minutes; check the log of the runner container at that point.
Monitoring
Section titled “Monitoring”kubectl get runnershows only names and ages.status.targetSizeholds the number of runner pods the controller aims for, and every change of it is aScaleUporScaleDownevent.- Informational events of the
RunnerincludeRegisterRunner,RollWorker,RollDeferred,DrainWorker,CachePVCExpired,CachePVCExpand,RollCacheServer, andCreate...andDelete...for every object the controller creates or deletes. - The controller’s Deployment has a memory limit of 128Mi. Its memory use grows with the number of pods, Secrets, ConfigMaps and volumes in the cluster, so on a large cluster raise the limit if the controller is terminated for exceeding its memory limit.
Upgrading
Section titled “Upgrading”Before upgrading, check that your Runner resources meet the requirements on these pages, in particular the minimum runner image version listed in Backends: runner pods with an older image may not work after the upgrade.
Apply the manifest of the new version, downloaded and verified as in Installing the controller:
kubectl apply -f runner-controller-v<version>-install.yaml- The manifest updates the custom resource definition and the controller together.
- Changes you made to the manifest, such as a mirrored controller image or a higher memory limit, must be made again to the manifest of the new version.
- When a new version generates the runner pods differently, it rebuilds them as for any configuration change, without interrupting jobs.
Runner images are upgraded by changing image in each Runner.
Uninstalling
Section titled “Uninstalling”Delete the Runner resources first. Their runner pods, job pods, volumes, Secrets and other objects are deleted with them:
kubectl -n <namespace> delete runner <name>kubectl -n <namespace> get pvc # wait until the volumes are goneThen delete the controller with the manifest of the installed version:
kubectl delete -f runner-controller-v<version>-install.yamlDeleting the Runner resources first lets you check that everything they created is gone. Delete a Runner while it runs no jobs: with the Docker-in-Docker backend its runner pods have their shutdownTimeout to finish them, but with the Kubernetes backend the runners lose their permissions together with the Runner, and their jobs fail.
Whether deleting a volume also frees its storage depends on the reclaim policy of the StorageClass. Under Retain, the persistent volumes stay behind as Released, and must be deleted by hand. storage.reclaimPolicy: Delete avoids that when set from the start.
The runners stay in the runner list of Gitea as offline runners, and can be deleted there.
Air-gapped clusters
Section titled “Air-gapped clusters”Images
Section titled “Images”Mirror these images to an internal registry:
-
The controller,
commitgo/runner-controller:v<version>: change theimageof the Deployment in the installation manifest before applying it. -
The runner, for example
gitea/runner:6.0.0-dind, which also runs the built-in cache server. Set it inimage, with credentials inimagePullSecrets, which sits at the top ofspec, not underruntime:imagePullSecrets:- name: <registry credentials>runtime:image: <internal registry>/runner:6.0.0-dind -
The job images, which the labels point to:
runtime:labels:ubuntu-latest: docker://<internal registry>/runner-images:ubuntu-latestWith the Docker-in-Docker backend, the Docker daemon in the runner pod pulls them with the credentials described in Private job images. With the Kubernetes backend, the job pods pull them with the
imagePullSecretsof the job pod template.
Actions
Section titled “Actions”Actions such as actions/checkout@v4 are downloaded from https://github.com by default, which fails without internet access. Mirror the actions the workflows use to the Gitea instance, and have them resolved there as described in the Gitea documentation, Where will the runner download scripts when using actions. This is a setting of Gitea and of the workflows, not of the Runner.