webnugget

Autoscaling Gitlab-CI builds on preemptible Google-Cloud instances

I needed a cost effective way to use Gitlab-CI without running multiple workstations or server 24/7 to process my CI Builds.

Currently I have a self hosted Version of Gitlab running on a dedicated Host. To shutdown my worker-nodes I went with the following strategy.

  • create a small instance on GCE to run my autoscaling runner and build-cache.
  • connect it to my gitlab host
  • spawn workers for builds on GCE as needed

This allows me to get small idle costs and fast machines for builds when I need them.

Letā€™s see how we can get this working.

1. Create a Google-Cloud Account

If you donā€™t already have a Google-Cloud Account you can register one here. Then head over to your Console and create a new project for our Gitlab-CI-Builds. create new project

2. Install gcloud for your OS

Download and install the gcloud tools to spin up vmā€™s from commandline. https://cloud.google.com/sdk/docs/

run gcloud initin your terminal to setup gcloud tools with your google-account. Answer the dialog questions and you are set.

3. Create your Autoscale-Runner VM

After you setup your gcloud tools you can run the following command to launch a VM that will be used for running our Docker-Machine, Cache (Minio) and Docker Registry Proxy.

gcloud compute --project "gitlab-ci-demo" instances create "gitlab-runner" --zone "europe-west1-c" --machine-type "f1-micro" --scopes "https://www.googleapis.com/auth/compute" --image-family "ubuntu-1604-lts" --image-project "ubuntu-os-cloud" --boot-disk-size "30"

This command will spawn a small compute instance called ā€œgitlab-runnerā€ in ā€œeurope-westā€ with permissions to create new instances. It will run the latest ubuntu lts version and has a disksize of 30GB.

If you need more diskspace adjust the above parameter as needed.

In Case you are running your VM in one of the US-Zones it may be possible to qualify for the always free-tier which letā€™s you run a f1-micro instance at no cost.

4. Install Components VM

Ok, letā€™s install Docker and Docker-Machine on the new instance. Login to the VM with gcloud compute ssh gitlab-runner it will create a new ssh-keypair for you and connect you to the instance.

4.1 Install Docker

First add the GPG-KEY

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

then add Docker-Repository

sudo apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-xenial main'

and install Docker

sudo apt-get update && sudo apt-get install -y docker-engine

4.2 Install Docker-Machine

Install Docker-Maschine with the following command:

curl -L https://github.com/docker/machine/releases/download/v0.10.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

This will download Docker-Machine and make it executable.

4.3 Install Gitlab-CI-Multirunner

Letā€™s install Gitlab-CI-Multi-Runner by running:

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash && sudo apt-get install gitlab-ci-multi-runner

4.4 Setup Google Container Registry as Dockerhub Proxy

To prevent fetching Docker Images from Dockerhub every time we start a build with Gitlab-CI you can setup gcr.io as registry.

To modify the docker settings stored in /etc/docker/daemon.json you need to get root privileges.

sudo -i #get root privileges
echo '{ "registry-mirrors": ["https://mirror.gcr.io"]}' > /etc/docker/daemon.json && service docker restart && docker system info
exit #leave root

You should see your docker system info printed to console containing the mirror registry.

4.5 Install Minio

Because Gitlab-CI-Multirunner only supports caching on S3 and you donā€™t want to transfer all the cached data from buildsteps between Amazon and Google you need to setup an alternative. This is where Minio enters the stage as it provides a S3 compliant solution which you can host yourself.

sudo docker run -it --restart always -p 9000:9000 -v /srv/minio/config:/root/.minio -v /srv/minio/files:/export  --name minio minio/minio:latest server /export

This command starts Minio inside a Docker-container and mounts files and config to /srv/minio on the host system. It also exposes port 9000 for connections from our ci-runners and generates random AccessKey and SecretKey. Take a note of these Keys because you will need them when setting up Gitlab-Runner.

5. Configure Gitlab-CI-Multi-Runner

Ok finally you got all your components installed, now itā€™s time to register and configure the autoscaling-runner.

gitlab-ci-multirunner register

This is how the register dialog looks likeā€¦

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://<your gitlab instance url>/ci
Please enter the gitlab-ci token for this runner:
<your gitlab runner token>
Please enter the gitlab-ci description for this runner:
[gitlab-runner]: google-autoscale-runner
Please enter the gitlab-ci tags for this runner (comma separated):

Whether to lock Runner to current project [true/false]:
[false]:
Registering runner... succeeded                     
Please enter the executor: ssh, docker-ssh+machine, kubernetes, shell, virtualbox, docker+machine, docker, docker-ssh, parallels:
docker+machine
Please enter the default Docker image (e.g. ruby:2.1):
node:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded

Insert your gitlab url and your Gitlab-Runner token, which can be obtained in the admin area under runners. Give your runner a name and if you want to, specify some tags for this runner, for example ā€œautoscaleā€ . When asked for the executor enter docker+machine. If you want to change the default image you can do so in the last step, I went for node:latest as default.

Now you should see a new shared runner within your Gitlab.

The last step is to configure the runner for autoscaling. Open the runner config file with:

sudo nano /etc/gitlab-runner/config.toml

And adjust it to your needs. Start with updating the concurrent runners. This value is the maximum of simultaneously spawned runners.

5.1 [runners.cache]

In the runner.cache section of config.toml you can configure the settings for your minio cache. Add your minio credentials, a bucket-name to save cache files and specify flags for insecure and shared. You need to add the insecure flag because we did not add certificates. If you expose your minio-instance to public you have to add these.

 [runners.cache]
    Type = "s3"
    ServerAddress = "gitlab-runner:9000" #your minio-instance
    AccessKey = "<your minio accesskey>" # minio AccessKey
    SecretKey = "<your minio secret>" # minio Secret
    BucketName = "cache" 
    Insecure = true
    Shared = true

5.2 [runners.machine]

In the runners.machine section you can specify IdleCount (number of runners always available) and IdleTime. Set the IdleTime to something higher than 900 seconds because google will charge each spawned runner for a minimum of 10 minutes.

Update MachineOptions with params for Docker-Machine you would like to use. You need to add the google-project, a machine type and an image you want to run. In the example below i choose the latest coreos-stable, because it has docker already built in. Configure some tags for your runners if you like, set them to be preemtible (much cheaper than normal instances) because the runners are removed anyway. Add a zone and configure docker-machine to use internal ips for communication between runners and docker-machine.

[runners.machine]
    IdleCount = 0
    IdleTime = 600
    MachineDriver = "google"
    MachineName = "auto-scale-runner-%s"
    MachineOptions = [
      "google-project=gitlab-ci-demo",
      "google-machine-type=g1-small",
      "google-machine-image=coreos-cloud/global/images/family/coreos-stable",
      "google-tags=gitlab-ci-slave",
      "google-preemptible=true",
      "google-zone=europe-west1-c",
      "google-use-internal-ip=true"
    ]

5.3 complete config.toml

For Reference youā€™ll find a config.toml below.

concurrent = 5
check_interval = 0
[[runners]]
  name = "autoscale-runner-google"
  url = "https://<gitlabinstance>/ci"
  token = "xxxxxxxxxxxxxxxxxxxxxx"
  executor = "docker+machine"
  [runners.docker]
    tls_verify = false
    image = "node:latest"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    Type = "s3"
    ServerAddress = "gitlab-runner:9000"
    AccessKey = "<your minio accesskey>"
    SecretKey = "<your minio secret>"
    BucketName = "cache"
    Insecure = true
    Shared = true
  [runners.machine]
    IdleCount = 0
    IdleTime = 600
    MachineDriver = "google"
    MachineName = "auto-scale-runner-%s"
    MachineOptions = [
      "google-project=gitlab-ci-demo",
      "google-machine-type=g1-small",
      "google-machine-image=coreos-cloud/global/images/family/coreos-stable",
      "google-tags=gitlab-ci-slave",
      "google-preemptible=true",
      "google-zone=europe-west1-c",
      "google-use-internal-ip=true"
    ]

If you modify the runner config, gitlab-ci-multirunner will pick up changes automaticly.

Thatā€™s it

I hope you got everything setup and are working on a scalable and less expensive CI-Solution now. If you have any Ideas how this setup or post could e improved please let me know.

In case this post was useful feel free to share it with others.

References & further reading

  • https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/autoscale.md
  • https://docs.docker.com/machine/drivers/gce/#options
  • https://docs.minio.io/
  • https://docs.gitlab.com/runner/install/
  • https://docs.docker.com/engine/installation/linux/ubuntu/#install-using-the-repository
  • https://cloud.google.com/sdk/gcloud/reference/compute/instances/create

Contact