Skip to content
Try Gitea Cloud ☁️ for 30 days → Accelerate your Development & Deploys!
Support

Operations

The controller reports problems as events on the Runner, so start with:

Terminal window
kubectl -n <namespace> describe runner <name>
kubectl -n <namespace> get runner,pod,pvc
kubectl -n <namespace> describe pod <pod>
kubectl -n runner-controller-system logs deploy/runner-controller-controller-manager

The container of the runner in a runner pod is named runner: kubectl -n <namespace> logs <pod> -c runner shows what the runner does.

While a Runner keeps failing, the controller retries it with growing delays. Editing the Runner triggers an immediate retry.

Event Cause and fix
RuntimeNotSet The Runner sets neither runtime nor kubernetesRuntime, which only happens to Runner resources created before upgrading to 0.7.10. The controller stops acting on it. Add one of them, with its labels.
InspectGiteaFailed with status 403 The license of Gitea Enterprise is missing or invalid. The controller creates no runner pods until it is fixed.
InspectGiteaFailed with status 401 The registration token was reset or deleted in Gitea. Update the Secret of registrationTokenSecretRef (or registrationToken) with the current token.
InspectGiteaFailed with status 404 instanceUrl does not point to a valid Gitea Enterprise instance, see Gitea URL.
InspectGiteaFailed without a status The controller cannot reach Gitea: check instanceUrl, the network between the controller and Gitea, and the certificate, see Self-signed certificates.
RegistrationTokenUnavailable The Secret of registrationTokenSecretRef, or its key, is missing or empty, or the value has leading or trailing whitespace, see Registration token.
RegisterRunnerFailed The controller cannot register a runner with Gitea. Check instanceUrl, the token, and the network between the controller and Gitea.
BuildRunnerWorkerFailed The Runner refers to something that is missing or invalid, named in the message: a Secret or ConfigMap of runnerEnv or runnerEnvFrom (runner environment), the S3 keys (cache S3 credentials), the external cache key (cache server external secret), a volumeName that is used twice or by the controller (is taken, is the controller's), an init container (runnerInitContainers), or a runner.file set in your own configuration, which should be left unset. Until it is fixed, the controller stops adjusting the runner pods of this Runner: it does not scale or apply changes, and the runner pods already running keep running. It is only checked while runner pods are needed.
ApplyRunnerWorkerFailed with violates PodSecurity The namespace does not allow the runner pods: privileged ones with the Docker-in-Docker backend, or init containers that do not meet restricted with the Kubernetes backend.
ApplyRunnerWorkerFailed with exists but is not managed by this Runner Kubernetes backend: a ServiceAccount or RoleBinding of the same name exists, see Permissions.
InvalidJobPodTemplate Kubernetes backend: the job pod template is invalid, and the controller stops acting on the Runner until it is fixed, see Invalid templates.
RunnerInitContainerFailed An init container failed or cannot start. The event has the exit code and the end of its log, kubectl -n <namespace> logs <pod> -c <init container> shows the whole log.
RunnerRegisteredItself The runner did not use the registration the controller made, or another runner of the same name is online in the same scope. Check the log of the runner container.
RunnerNotOnline The runner has run for 3 minutes without coming online in Gitea. Check the log of the runner container. If it tries to register, for example with token is empty, the image version does not work with the controller.
DestroyWorker A runner pod did not come online and was created again with empty caches, see below.
GracePeriodTooSmall terminationGracePeriodSeconds is below shutdownTimeout, so jobs can be terminated before they finish.
JobTmpfsMemoryTooSmall The memory request or limit cannot hold jobTmpfs, see tmpfs memory.
CachePVCClassChangeIgnored, CachePVCShrinkIgnored, CachePVCExpandUnsupported A storage change that needs a new volume, which allowDeleteCachePVC does not allow, see Changing StorageClass or size.
Symptom Cause and fix
Creating or changing a Runner fails with one of runtime and kubernetesRuntime must be set or labels: Required value Set one of runtime and kubernetesRuntime, with the labels your workflows use in runs-on, see Labels.
Volumes stay Pending No StorageClass of that name, or it does not provision dynamically. className is standard by default, see Volumes of a runner pod. kubectl describe pvc shows why.
Pods Pending with volume node affinity conflict No node can run the pod where its volume is, see Storage requirements. If the volumes are bound to a node that no longer exists, delete them and the pod: kubectl -n <namespace> delete pvc <runner pod>-action-cache <runner pod>-cache --wait=false (-cache with Docker-in-Docker only), then kubectl -n <namespace> delete pod <runner pod>. The controller creates the pod again with new, empty volumes.
Pods Pending with no matching node nodeSelector matches no node, a taint is not tolerated, or the cluster is full.
Scaling up takes minutes This is expected when scaling up from zero: a cold start includes a new node, if needed, and the image pull. Use minSize to keep runners available.
Runner pods are not rebuilt after a change, with RollDeferred events They run jobs, and are rebuilt once the jobs are done, see How configuration changes are applied.
Jobs fail to connect to Docker Docker-in-Docker backend: the runner image is not a -dind variant.
With a proxy, the runners do not come online, the workflow cache stops working, or job pods cannot be created NO_PROXY misses an address, see Proxy.
The workflow cache always misses with S3 The log of the runner container shows errors with sync S3 cache or upload cache to S3: check the endpoint, region, keys and permissions, including the bucket policy.
The workflow cache always misses with the built-in cache server The cluster domain is not cluster.local: set cacheServer.host, see Built-in cache server.
forbidden errors for system:serviceaccount:runner-controller-system:runner-controller-controller-manager, in the controller log or in warning events The controller lacks a permission, see Missing permissions.
Kubernetes backend: jobs fail with pods is forbidden The ServiceAccount of the runner lacks permissions, see Permissions.
Kubernetes backend: jobs fail with violates PodSecurity The job pods do not meet the level of the namespace, see Pod Security level.
Kubernetes backend: jobs fail to download actions, with permission denied in the log of the runner container The StorageClass does not support fsGroup, see Storage. With your own configuration, host.workdir_parent is missing, see Using your own runner configuration.
Kubernetes backend: job pods are evicted for disk pressure or ephemeral-storage The node disk cannot hold the job images and workspaces, see Storage.
Kubernetes backend: jobs wait with Waiting for pod ... to be scheduled No node fits the resources of the job container or the scheduling of the job pod template.
Kubernetes backend: the workflow cache times out The job pods cannot reach port 8088 of their runner pod, check the NetworkPolicies.

When the runner of a runner pod has not been online in Gitea for 5 minutes, and the pod started at least 15 minutes ago, image pulls and init containers included, the controller deletes the pod with its configuration and its cache volumes, reports a DestroyWorker warning, and creates it again. The registration is kept, unless Gitea no longer lists that runner, in which case it is registered anew. Pods that have not started, such as Pending ones, are left alone.

This also applies to a runner that came online and went offline later. Before that, the controller reports RunnerNotOnline after 3 minutes; check the log of the runner container at that point.

  • kubectl get runner shows only names and ages. status.targetSize holds the number of runner pods the controller aims for, and every change of it is a ScaleUp or ScaleDown event.
  • Informational events of the Runner include RegisterRunner, RollWorker, RollDeferred, DrainWorker, CachePVCExpired, CachePVCExpand, RollCacheServer, and Create... and Delete... for every object the controller creates or deletes.
  • The controller’s Deployment has a memory limit of 128Mi. Its memory use grows with the number of pods, Secrets, ConfigMaps and volumes in the cluster, so on a large cluster raise the limit if the controller is terminated for exceeding its memory limit.

Before upgrading, check that your Runner resources meet the requirements on these pages, in particular the minimum runner image version listed in Backends: runner pods with an older image may not work after the upgrade.

Apply the manifest of the new version, downloaded and verified as in Installing the controller:

Terminal window
kubectl apply -f runner-controller-v<version>-install.yaml
  • The manifest updates the custom resource definition and the controller together.
  • Changes you made to the manifest, such as a mirrored controller image or a higher memory limit, must be made again to the manifest of the new version.
  • When a new version generates the runner pods differently, it rebuilds them as for any configuration change, without interrupting jobs.

Runner images are upgraded by changing image in each Runner.

Delete the Runner resources first. Their runner pods, job pods, volumes, Secrets and other objects are deleted with them:

Terminal window
kubectl -n <namespace> delete runner <name>
kubectl -n <namespace> get pvc # wait until the volumes are gone

Then delete the controller with the manifest of the installed version:

Terminal window
kubectl delete -f runner-controller-v<version>-install.yaml

Deleting the Runner resources first lets you check that everything they created is gone. Delete a Runner while it runs no jobs: with the Docker-in-Docker backend its runner pods have their shutdownTimeout to finish them, but with the Kubernetes backend the runners lose their permissions together with the Runner, and their jobs fail.

Whether deleting a volume also frees its storage depends on the reclaim policy of the StorageClass. Under Retain, the persistent volumes stay behind as Released, and must be deleted by hand. storage.reclaimPolicy: Delete avoids that when set from the start.

The runners stay in the runner list of Gitea as offline runners, and can be deleted there.

Mirror these images to an internal registry:

  1. The controller, commitgo/runner-controller:v<version>: change the image of the Deployment in the installation manifest before applying it.

  2. The runner, for example gitea/runner:6.0.0-dind, which also runs the built-in cache server. Set it in image, with credentials in imagePullSecrets, which sits at the top of spec, not under runtime:

    imagePullSecrets:
    - name: <registry credentials>
    runtime:
    image: <internal registry>/runner:6.0.0-dind
  3. The job images, which the labels point to:

    runtime:
    labels:
    ubuntu-latest: docker://<internal registry>/runner-images:ubuntu-latest

    With the Docker-in-Docker backend, the Docker daemon in the runner pod pulls them with the credentials described in Private job images. With the Kubernetes backend, the job pods pull them with the imagePullSecrets of the job pod template.

Actions such as actions/checkout@v4 are downloaded from https://github.com by default, which fails without internet access. Mirror the actions the workflows use to the Gitea instance, and have them resolved there as described in the Gitea documentation, Where will the runner download scripts when using actions. This is a setting of Gitea and of the workflows, not of the Runner.