GitLab Runner

Simplifying GitLab Runner Setup in a Kubernetes Cluster Using Terraform

Setting up a GitLab Runner in a Kubernetes cluster can significantly streamline your CI/CD pipeline by leveraging the scalability and flexibility of Kubernetes. In this article, I will guide you through a hassle-free setup process using Terraform. This approach minimizes configuration overhead and ensures a smooth deployment.

Kubernetes Executor is one of the executors for GitLab Runner, allowing it to run jobs on a Kubernetes cluster. This is particularly useful for scaling and managing workloads efficiently in a cloud-native environment.

How GitLab Runner Kubernetes Executor Works

  1. Runner Registration: First, you register the GitLab Runner with your GitLab instance and configure it to use the Kubernetes executor.
  2. Job Execution: When a job is triggered, the GitLab Runner interacts with the Kubernetes API to spin up a new pod that will run the job.
  3. Pod Lifecycle:
    • Pod Creation: A pod is created for each job, which includes one or more containers.
    • Job Execution: The job runs inside the containers of the pod.
    • Pod Deletion: Once the job is complete, the pod is automatically deleted.
Runner pods lifecycle

Benefits

  • Scalability: Automatically scale the number of runners based on the demand.
  • Isolation: Each job runs in a separate pod, ensuring clean environments for every job.
  • Resource Management: Kubernetes efficiently manages resources and schedules jobs.
  • Flexibility: Supports various environments and tools by using different container images.

Prerequisites

Before we start, better to have the following:

  • Basic knowledge of Kubernetes, GitLab, and Terraform.
  • A Kubernetes cluster.
  • A GitLab account.
  • Terraform installed on your machine.

Setting Up GitLab Runner with Terraform

I successfully set up a GitLab Runner on a Kubernetes cluster using Terraform, ensuring an efficient and scalable CI/CD pipeline. This setup allows for automated management of infrastructure as code, leveraging Terraform’s robust provisioning capabilities and Kubernetes’ orchestration strengths to streamline job execution and resource management within the cluster. This article explains the way I manage to automate the deployment of a GitLab Runner in a Kubernetes cluster. This includes creating necessary Kubernetes resources like secrets, config maps, persistent volumes, and deployments.

Terraform Script for GitLab Runner Setup

Below is the Terraform script used to set up the GitLab Runner. Each section of the script is explained in detail.

terraform {
  required_version = "1.8.3"
  required_providers {
    kubernetes = {
        version = "~>2.0"
    }
  }
}

provider "kubernetes" {
  config_path = "./config"
}

locals {
  label_name = "gitlab-runner"
}

variable "virtual_cluster_name" {
  type = string
  default = "gitlab-runners"
}

variable "docker_pull_secret" {
  type = string
  default = "docker.company.com"
}

resource "kubernetes_secret" "docker_pull_secret" {
  metadata {
    name = var.docker_pull_secret
    namespace = var.virtual_cluster_name
  }

  data = {
    ".dockerconfigjson" = "{\"auths\": {\"docker.company.com\": {\"username\": \"ci-username\", \"password\": \"ci-password\", \"email\": \"ci-user@mail.com\"}}}"
  }
  type = "kubernetes.io/dockerconfigjson"
}

resource "kubernetes_config_map" "gitlab_runner_config" {
  metadata {
    name = "gitlab-runner-config"
    namespace = var.virtual_cluster_name
    labels = {
      label = local.label_name
    }
  }

  data = {
    "config.toml" = templatefile("./runner-config-toml.tpl", 
    {
        name = "gitlab-runner"
        token = "my-gitlab-token"
        docker_pull_secret = var.docker_pull_secret
        namespace = var.virtual_cluster_name
    })
  }
}

resource "kubernetes_persistent_volume" "maven_local_volume" {
  metadata {
    name = "gitlab-runner-mvn-local-volume"
  }
  spec {
    access_modes = [ "ReadWriteOnce" ]
    capacity = {
      storage = "5Gi"
    }
    persistent_volume_source {
      host_path {
        path = "/mnt/.m2"
      }
    }
  }
}

resource "kubernetes_secret" "gitlab_token" {
  metadata {
    name = "gitlab-access-token"
    namespace = var.virtual_cluster_name
    labels = {
      label = local.label_name
    }
  }

  data = {
    GITLAB_CI_TOKEN = "my-gitlab-ci-token"
  }
}

resource "kubernetes_deployment" "gitlab_runner_deployment" {
  metadata {
    name = "gitlab-runner"
    namespace = var.virtual_cluster_name
  }
  spec {
    replicas = 1
    selector {
      match_labels = {
        label = local.label_name
      }
    }

    template {
      metadata {
        name = "gitlab-runner"
        namespace = var.virtual_cluster_name
        labels = {
          label = local.label_name
        }
      }
      spec {
        automount_service_account_token = true
        service_account_name = "gitlab-runner-admin"
        image_pull_secrets {
          name = var.docker_pull_secret
        }
        dns_policy = "None"
        dns_config {
          nameservers = [ "xx.xx.xx.xx" ]
          searches = [ "gitlab.company.com" ]
        }
        container {
          name = "gitlab-runner"
          image = "gitlab/gitlab-runner:v16.11.0"
          image_pull_policy = "Always"

          args = [ "run" ]

          volume_mount {
            mount_path = "/etc/gitlab-runner"
            name = "config"
          }
          volume_mount {
            mount_path = "/etc/ssl/certs"
            name = "cacerts"
            read_only = true
          }
        }

        restart_policy = "Always"
        
        volume {
          config_map {
            name = "gilab-runner-config"
          }
          name = "config"
        }
        volume {
            host_path {
                path = "/usr/share/ca-certificates/mozilla"
            }  
            name = "cacerts"
        }
      }
    }
  }
}

resource "kubernetes_service_account" "gitlab_runner_service_account" {
  metadata {
    name = "gitlab-runner-admin"
    namespace = var.virtual_cluster_name
  }
  image_pull_secret {
    name = "docker-secret"
  }
  secret {
    name = "gitlab-ci-token"
  }
  automount_service_account_token = true
}

resource "kubernetes_role" "gitlab_ci_role" {
  metadata {
    name = "gitlab-runner-role"
    namespace = var.virtual_cluster_name
    labels = {
      label = local.label_name
    }
  }
  rule {
    api_groups = [ "" ]
    resources = [ "" ]
    verbs = [ "" ]
  }
}

resource "kubernetes_role_binding" "gitlab_ci_role_binding" {
  metadata {
    name = "gitlab-runner-role-binding"
    namespace = var.virtual_cluster_name
  }
  subject {
    kind = "ServiceAccount"
    name = "gitlab-runner-admin"
    namespace = var.virtual_cluster_name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind = "Role"
    name = "gitlab-runner-role"
  }
}

runner-config-toml.tpl file as bellow. 

concurrent = 12

[[runners]]
    name = "${name}"
    url = "https://gitlab.company.com"
    token = "${token}"
    output_limit = 10000
    executor = "kubernetes"
    [runners.kubernetes]
        namespace = "${namespace}"
        privilaged = true
        image_pull_secrets = ["${docker_pull_secret}"]
        cpu_request = "1"
        memory_request = "1Gi"
        cpu_limit = "2"
        memory_limit = "2Gi"
        [runners.kubernetes.dns.config]
            nameservers = ["xx.xx.xx.xx"]
            searches = ["gitlab.company.com"]
        [[runners.kubernetes.host_aliases]]
            ip = "xx.xx.xx.xx"
            hostname = ["gitlab.company.com"]
        [[runners.kubernetes.host_aliases]]
            ip = "xx.xx.xx.xx"
            hostname = ["artifactory.company.com"]
        [[runners.kubernetes.volume.host_path]]
            name = "m2_dir"
            mount_path = "./m2"
            host_path = "/mnt/.m2"
            read_only = false

Deploying the GitLab Runner

Generate GitLab Personal Token for Runner use.

Generating a GitLab Personal Access Token (PAT) is a straightforward process that can enhance the security and functionality of your GitLab Runner deployment, especially if the runner tasks involve GitLab repository operations. Although a personal access token is not mandatory for setting up a GitLab Runner, it is highly recommended for tasks that require repository access.

To generate a PAT, navigate to your GitLab account settings, find the “Access Tokens” section, and create a new token with the necessary scopes such as ‘api’ or ‘read_repository’ based on your needs. Once generated, this token can be securely stored in Kubernetes secrets associated with your runner, ensuring that your GitLab Runner can perform operations on your repositories seamlessly and securely.

Generate GitLab Personal Access Token for GitLab Runner Use

Generate GitLab Runner Token

Creating a GitLab Runner Token is an essential step for registering your GitLab Runner with a GitLab instance. To generate this token, you need to navigate to your project’s settings in GitLab. Under “Settings,” select “CI / CD” and then find the “Runners” section. Here, click on “New runner” to initiate the token generation process. GitLab will provide you with a unique token specifically for your runner.

This token is crucial for the runner to authenticate and communicate with the GitLab instance. Once you have the token, you can use it during the runner registration process, typically through the GitLab Runner’s command line interface. This ensures that your runner is properly registered and able to execute CI/CD pipelines for your projects.

Create Runner to Obtain GitLab token

Tags in GitLab Runner are used to ensure that specific jobs run on particular runners. By assigning tags to a runner and using the same tags in your job definitions, you can control which jobs are executed by which runners. This is useful for directing jobs to runners with specific capabilities, such as certain software dependencies or hardware configurations. Tags help streamline your CI/CD process by organizing and managing job execution efficiently.

Timeouts in GitLab Runner define the maximum time a job can run before it is automatically terminated. Setting a timeout helps prevent jobs from running indefinitely, which can consume resources unnecessarily and block other jobs.

Generated GitLab Runner Token

Deploying the GitLab Runner using Terraform

When configuring your GitLab Runner, make sure to replace all placeholder tokens and credentials with your own. Follow these steps to deploy the GitLab Runner using the provided Terraform script and config.toml.tpl file:

  1. Ensure you have your Kubernetes configuration file at ./config.
  2. Save the Terraform script and runner-config-toml.tpl to same directory.
  3. Initialize Terraform:
terraform init
  1. Apply the Terraform script:
terraform apply

When you applying script shell will ask to apply or not. type yes and press enter key.

Verifying the Setup

To verify that the GitLab Runner is set up correctly:

  1. Check the deployment status:
kubectl get deployments -n gitlab-runners

This command output will show the deployments under gitlab-runners namespace. Verify and make sure gitlab-runner deployment is available there.

  1. Ensure the GitLab Runner pod is running:
kubectl get pods -n gitlab-runners
  1. Verify the runner is registered in your GitLab instance under the CI/CD settings.

Detail Explanation of Terraform Script

terraform {
  required_version = "1.8.3"
  required_providers {
    kubernetes = {
        version = "~>2.0"
    }
  }
}

provider "kubernetes" {
  config_path = "./config"
}

1. terraform block: This block specifies the Terraform version and the required provider versions for the script. The required_version ensures compatibility with Terraform version 1.8.3, and the required_providers section specifies the Kubernetes provider version to be used, ensuring that the script runs with the appropriate versions and dependencies.

2. provider "kubernetes" block: This block configures the Kubernetes provider, indicating where to find the kubeconfig file. The config_path points to a local file (./config) that contains the necessary configuration to connect to the Kubernetes cluster. This allows Terraform to authenticate and interact with the cluster using the specified credentials and context.

locals {
  label_name = "gitlab-runner"
}

variable "virtual_cluster_name" {
  type = string
  default = "gitlab-runners"
}

3. Locals Block: This block defines a local variable label_name with the value “gitlab-runner”. Local variables in Terraform help manage repeated values or complex expressions more easily. Here, it centralizes the labeling used throughout the script, ensuring consistency and simplifying updates if the label needs to change.

4. Variable Block: These blocks declare variables that can be customized without modifying the script directly. virtual_cluster_name specifies the namespace within the Kubernetes cluster where resources will be deployed. 

variable "docker_pull_secret" {
  type = string
  default = "docker.company.com"
}

resource "kubernetes_secret" "docker_pull_secret" {
  metadata {
    name = var.docker_pull_secret
    namespace = var.virtual_cluster_name
  }

  data = {
    ".dockerconfigjson" = "{\"auths\": {\"docker.company.com\": {\"username\": \"ci-username\", \"password\": \"ci-password\", \"email\": \"ci-user@mail.com\"}}}"
  }
  type = "kubernetes.io/dockerconfigjson"
}

5. Variable Block: This block defines an input variable named docker_pull_secret.

6. Resource Block: This block creates a Kubernetes secret for Docker registry authentication. The secret stores credentials (username, password, and email) in a .dockerconfigjson format. This secret allows the GitLab Runner to authenticate with the Docker registry specified by docker_pull_secret, enabling it to pull necessary Docker images securely.

resource "kubernetes_config_map" "gitlab_runner_config" {
  metadata {
    name = "gitlab-runner-config"
    namespace = var.virtual_cluster_name
    labels = {
      label = local.label_name
    }
  }

  data = {
    "config.toml" = templatefile("./runner-config-toml.tpl", 
    {
        name = "gitlab-runner"
        token = "my-gitlab-token"
        docker_pull_secret = var.docker_pull_secret
        namespace = var.virtual_cluster_name
    })
  }
}

7. Resource Block: This block defines a ConfigMap to store the GitLab Runner configuration, which is essential for the runner’s operation. The templatefile function dynamically generates the config.toml file using a template and variables. This ConfigMap centralizes the runner configuration, making it easily manageable and ensuring the runner has the necessary settings to connect to GitLab and use the Docker pull secret.

resource "kubernetes_persistent_volume" "maven_local_volume" {
  metadata {
    name = "gitlab-runner-mvn-local-volume"
  }
  spec {
    access_modes = [ "ReadWriteOnce" ]
    capacity = {
      storage = "5Gi"
    }
    persistent_volume_source {
      host_path {
        path = "/mnt/.m2"
      }
    }
  }
}

8. Resource Block: This block creates a PersistentVolume (PV) for caching Maven artifacts, which is particularly useful for improving build performance. By storing Maven dependencies in a persistent volume, the GitLab Runner can reuse these artifacts across multiple builds, reducing build times and network usage. The PV is configured with a host path (/mnt/.m2) and a storage capacity of 5Gi, making it accessible to the runner.

resource "kubernetes_secret" "gitlab_token" {
  metadata {
    name = "gitlab-access-token"
    namespace = var.virtual_cluster_name
    labels = {
      label = local.label_name
    }
  }

  data = {
    GITLAB_CI_TOKEN = "my-gitlab-ci-token"
  }
}

9. Resource Block: This block creates a Kubernetes secret to securely store the GitLab CI token, which is necessary for authenticating with the GitLab CI/CD system. By storing the token in a secret, it ensures that sensitive information is kept secure and is only accessible to the GitLab Runner and other authorized components.

resource "kubernetes_deployment" "gitlab_runner_deployment" {
  metadata {
    name = "gitlab-runner"
    namespace = var.virtual_cluster_name
  }
  spec {
    replicas = 1
    selector {
      match_labels = {
        label = local.label_name
      }
    }

    template {
      metadata {
        name = "gitlab-runner"
        namespace = var.virtual_cluster_name
        labels = {
          label = local.label_name
        }
      }
      spec {
        automount_service_account_token = true
        service_account_name = "gitlab-runner-admin"
        image_pull_secrets {
          name = var.docker_pull_secret
        }
        dns_policy = "None"
        dns_config {
          nameservers = [ "xx.xx.xx.xx" ]
          searches = [ "gitlab.company.com" ]
        }
        container {
          name = "gitlab-runner"
          image = "gitlab/gitlab-runner:v16.11.0"
          image_pull_policy = "Always"

          args = [ "run" ]

          volume_mount {
            mount_path = "/etc/gitlab-runner"
            name = "config"
          }
          volume_mount {
            mount_path = "/etc/ssl/certs"
            name = "cacerts"
            read_only = true
          }
        }

        restart_policy = "Always"
        
        volume {
          config_map {
            name = "gilab-runner-config"
          }
          name = "config"
        }
        volume {
            host_path {
                path = "/usr/share/ca-certificates/mozilla"
            }  
            name = "cacerts"
        }
      }
    }
  }
}

10. Resource Deployment Block: This block defines the deployment for the GitLab Runner. It specifies the deployment metadata, including the name and namespace. The deployment is configured to run a single replica of the GitLab Runner. The container configuration includes the Docker image, pull policy, and arguments to start the runner. Volume mounts are used to attach the ConfigMap and SSL certificates to the container, ensuring the runner has the necessary configuration and can establish secure connections. DNS settings are specified to ensure the runner can resolve the GitLab DNS, which is crucial in restricted networks.

resource "kubernetes_service_account" "gitlab_runner_service_account" {
  metadata {
    name = "gitlab-runner-admin"
    namespace = var.virtual_cluster_name
  }
  image_pull_secret {
    name = "docker-secret"
  }
  secret {
    name = "gitlab-ci-token"
  }
  automount_service_account_token = true
}

11. Resource Service Account: This block creates a service account for the GitLab Runner, providing it with necessary permissions and secrets. The service account is linked to the Docker pull secret and the GitLab CI token, enabling the runner to authenticate with the Docker registry and GitLab. The automount_service_account_token is set to true, ensuring that the service account token is automatically mounted in the runner pod.

resource "kubernetes_role" "gitlab_ci_role" {
  metadata {
    name = "gitlab-runner-role"
    namespace = var.virtual_cluster_name
    labels = {
      label = local.label_name
    }
  }
  rule {
    api_groups = [ "" ]
    resources = [ "" ]
    verbs = [ "" ]
  }
}

resource "kubernetes_role_binding" "gitlab_ci_role_binding" {
  metadata {
    name = "gitlab-runner-role-binding"
    namespace = var.virtual_cluster_name
  }
  subject {
    kind = "ServiceAccount"
    name = "gitlab-runner-admin"
    namespace = var.virtual_cluster_name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind = "Role"
    name = "gitlab-runner-role"
  }
}

12. Resource Kubernetes Role Binding: These blocks set up a Role and Role Binding to grant the necessary permissions to the GitLab Runner service account. The Role defines a set of permissions, while the Role Binding associates the Role with the service account (gitlab-runner-admin). This setup ensures that the service account has the required access to Kubernetes resources within the specified namespace, allowing the runner to perform.

We invite you to try out this GitLab Runner setup using Terraform. This script is designed to streamline the deployment process, ensuring a smooth and efficient setup. By following the instructions provided, you’ll be able to deploy a GitLab Runner with persistent volume support for Maven caching, DNS configuration, and secure certificate usage.

You can find the complete script and detailed instructions on my GitHub repository: GitLab Runner Terraform Setup

I’d love to hear about your experience with this setup. Feel free to share your feedback, issues, or any improvements you suggest. Your input will help me enhance the deployment process and ensure it meets your needs effectively.

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *