Download and install the gcloud tools to spin up vmās from commandline. https://cloud.google.com/sdk/docs/
run gcloud init
in your terminal to setup gcloud tools with your google-account. Answer the dialog questions and you are set.
After you setup your gcloud tools you can run the following command to launch a VM that will be used for running our Docker-Machine, Cache (Minio) and Docker Registry Proxy.
gcloud compute --project "gitlab-ci-demo" instances create "gitlab-runner" --zone "europe-west1-c" --machine-type "f1-micro" --scopes "https://www.googleapis.com/auth/compute" --image-family "ubuntu-1604-lts" --image-project "ubuntu-os-cloud" --boot-disk-size "30"
This command will spawn a small compute instance called āgitlab-runnerā in āeurope-westā with permissions to create new instances. It will run the latest ubuntu lts version and has a disksize of 30GB.
If you need more diskspace adjust the above parameter as needed.
In Case you are running your VM in one of the US-Zones it may be possible to qualify for the always free-tier which letās you run a f1-micro instance at no cost.
Ok, letās install Docker and Docker-Machine on the new instance. Login to the VM with gcloud compute ssh gitlab-runner
it will create a new ssh-keypair for you and connect you to the instance.
First add the GPG-KEY
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
then add Docker-Repository
sudo apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-xenial main'
and install Docker
sudo apt-get update && sudo apt-get install -y docker-engine
Install Docker-Maschine with the following command:
curl -L https://github.com/docker/machine/releases/download/v0.10.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine
This will download Docker-Machine and make it executable.
Letās install Gitlab-CI-Multi-Runner by running:
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash && sudo apt-get install gitlab-ci-multi-runner
To prevent fetching Docker Images from Dockerhub every time we start a build with Gitlab-CI you can setup gcr.io as registry.
To modify the docker settings stored in /etc/docker/daemon.json you need to get root privileges.
sudo -i #get root privileges
echo '{ "registry-mirrors": ["https://mirror.gcr.io"]}' > /etc/docker/daemon.json && service docker restart && docker system info
exit #leave root
You should see your docker system info printed to console containing the mirror registry.
Because Gitlab-CI-Multirunner only supports caching on S3 and you donāt want to transfer all the cached data from buildsteps between Amazon and Google you need to setup an alternative. This is where Minio enters the stage as it provides a S3 compliant solution which you can host yourself.
sudo docker run -it --restart always -p 9000:9000 -v /srv/minio/config:/root/.minio -v /srv/minio/files:/export --name minio minio/minio:latest server /export
This command starts Minio inside a Docker-container and mounts files and config to /srv/minio on the host system. It also exposes port 9000 for connections from our ci-runners and generates random AccessKey and SecretKey. Take a note of these Keys because you will need them when setting up Gitlab-Runner.
Ok finally you got all your components installed, now itās time to register and configure the autoscaling-runner.
gitlab-ci-multirunner register
This is how the register dialog looks likeā¦
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://<your gitlab instance url>/ci
Please enter the gitlab-ci token for this runner:
<your gitlab runner token>
Please enter the gitlab-ci description for this runner:
[gitlab-runner]: google-autoscale-runner
Please enter the gitlab-ci tags for this runner (comma separated):
Whether to lock Runner to current project [true/false]:
[false]:
Registering runner... succeeded
Please enter the executor: ssh, docker-ssh+machine, kubernetes, shell, virtualbox, docker+machine, docker, docker-ssh, parallels:
docker+machine
Please enter the default Docker image (e.g. ruby:2.1):
node:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded
Insert your gitlab url and your Gitlab-Runner token, which can be obtained in the admin area under runners. Give your runner a name and if you want to, specify some tags for this runner, for example āautoscaleā . When asked for the executor enter docker+machine. If you want to change the default image you can do so in the last step, I went for node:latest as default.
Now you should see a new shared runner within your Gitlab.
The last step is to configure the runner for autoscaling. Open the runner config file with:
sudo nano /etc/gitlab-runner/config.toml
And adjust it to your needs. Start with updating the concurrent runners. This value is the maximum of simultaneously spawned runners.
In the runner.cache section of config.toml you can configure the settings for your minio cache. Add your minio credentials, a bucket-name to save cache files and specify flags for insecure and shared. You need to add the insecure flag because we did not add certificates. If you expose your minio-instance to public you have to add these.
[runners.cache]
Type = "s3"
ServerAddress = "gitlab-runner:9000" #your minio-instance
AccessKey = "<your minio accesskey>" # minio AccessKey
SecretKey = "<your minio secret>" # minio Secret
BucketName = "cache"
Insecure = true
Shared = true
In the runners.machine section you can specify IdleCount (number of runners always available) and IdleTime. Set the IdleTime to something higher than 900 seconds because google will charge each spawned runner for a minimum of 10 minutes.
Update MachineOptions with params for Docker-Machine you would like to use. You need to add the google-project, a machine type and an image you want to run. In the example below i choose the latest coreos-stable, because it has docker already built in. Configure some tags for your runners if you like, set them to be preemtible (much cheaper than normal instances) because the runners are removed anyway. Add a zone and configure docker-machine to use internal ips for communication between runners and docker-machine.
[runners.machine]
IdleCount = 0
IdleTime = 600
MachineDriver = "google"
MachineName = "auto-scale-runner-%s"
MachineOptions = [
"google-project=gitlab-ci-demo",
"google-machine-type=g1-small",
"google-machine-image=coreos-cloud/global/images/family/coreos-stable",
"google-tags=gitlab-ci-slave",
"google-preemptible=true",
"google-zone=europe-west1-c",
"google-use-internal-ip=true"
]
For Reference youāll find a config.toml below.
concurrent = 5
check_interval = 0
[[runners]]
name = "autoscale-runner-google"
url = "https://<gitlabinstance>/ci"
token = "xxxxxxxxxxxxxxxxxxxxxx"
executor = "docker+machine"
[runners.docker]
tls_verify = false
image = "node:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Type = "s3"
ServerAddress = "gitlab-runner:9000"
AccessKey = "<your minio accesskey>"
SecretKey = "<your minio secret>"
BucketName = "cache"
Insecure = true
Shared = true
[runners.machine]
IdleCount = 0
IdleTime = 600
MachineDriver = "google"
MachineName = "auto-scale-runner-%s"
MachineOptions = [
"google-project=gitlab-ci-demo",
"google-machine-type=g1-small",
"google-machine-image=coreos-cloud/global/images/family/coreos-stable",
"google-tags=gitlab-ci-slave",
"google-preemptible=true",
"google-zone=europe-west1-c",
"google-use-internal-ip=true"
]
If you modify the runner config, gitlab-ci-multirunner will pick up changes automaticly.
I hope you got everything setup and are working on a scalable and less expensive CI-Solution now. If you have any Ideas how this setup or post could e improved please let me know.
In case this post was useful feel free to share it with others.
References & further reading