Auto scale GitHub Actions Runner with Google Cloud Platform

I used many CI services e.g: Gitlab CI, Circle CI, Travis CI and recently is GitHub Actions and I think Gitlab CI is the best one (just my opinion). But my company is using Travis CI because it’s cheapest but in Travis CI does not support self-hosted runner. Circle CI is too expensive. And Gitlab is supported self-host runner but we using GitHub and almost our community is using GitHub so we can’t migrate to Gitlab and the last candidate is GitHub Actions. Thanks God It’s supported self-hosted runner at 2019.

There are 2 levels to register runner is Repository Level and Organization Level. If you registered self-hosted runner at both level GitHub will choose in Repository level first. But in this article I just register self-hosted runner at Organization level.

The platform I’m using is Google Cloud Platform (GCP) and we will using Preemptible Virtual Machine (VM). Preemptible VM is spot instance and it will be deleted after 24h. You can find more information of preemptible vm in here.

The idea

• GitHub self-hosted runner just an agent that you need to install on your instance that you needs to execute workflows.
• Everytime a workflow is triggered, a check_run event is created. We can use this event to decide whether to scale a runner.
• There are 2 options to handler check_run event is using webhook at the Repository level and Organization level or using GitHub app. In this article I’m using GitHub app.
• How about scale down? After the workflow picked up a self-hosted runner and finished jobs and a check_run event will be created with status is completed . We can use this event to scale down instance.

GCP Service Account

• We need a service account can read and write compute instance.
• You can restrict policy of this service account with conditions when you create service account.

VM Image

• To speed up create instance phase. I’m using VM Images. You can find it in compute/storage/Images .
• Create folder actions-runner in /opt
• Follow GitHub create self-hosted runner instruction before step run deploy.sh then create a image from your current disk.

GitHub App

• Choose you app name
• Set the webhook url. It’s your app API. You make your own or you can use my github-spawner .
• The secret: This secret to verify webhook payload
• Repository permissions:
• Administration (read, write) to register runner.
• Checks (read) to get check_run event.
• Subscribe to events: [Check run]

GitHub-Spawner

• I created a simple project to handle check_run event and I called it is github-spawner this is the Source Code (vote a start if it helpful). The spawner just python server and contain a single API POST /api/v1/webhook to handle GitHub event.
• There are feel environments you need to implements here:

DEBUG=1 # Just for debug mode  
SECRET_KEY=very-secret # Your secret APP
ALLOWED_HOSTS=* # CORS
GITHUB_APP_ID=your-app-id
GITHUB_APP_SECRET=github-app-secret # Remember this is app secret when you create app. It's not client secret
GITHUB_APP_PEM=base64-app-private-key # This private key to create app token
GCP_ROBOT_CRED=base64-credential
GCP_PROJECT_ID=gcp-project-id
GCP_DEFAULT_REGION=your-region
GCP_DEFAULT_ZONE=your-timezone
GCP_SERVICE_ACC=service-account@project-id.com

VM Instance

• In GCP when you create a instance you can execute a script everytime the vm startup it called is startup-script .
• We can write a startup script to register the instance to GitHub Actions.

#!/bin/bash
user=runner
id -u $user &> /dev/null | sudo useradd $user
sudo chmod -R 777 /opt/actions-runner
sudo -u bash -c 'cd /opt/actions-runner; echo -ne "\n\n\n" | ./config.sh --url https://github.com/{org} --token {token}'

Basically this script will create a user and named runner . And run config.sh . Because GCP run startup script with sudo but config.sh cannot run with sudo permission.

Note: Make sure you registered a self-hosted runner and let it offline. Because when the check_run event created and the instance is creating. At this time GitHub will find the self-hosted runner and no self-hosted runner register. It will stop the workloads with error message is not self-hosted runner registered. So we need at least 1 self-hosted runner to holding the workloads until the new instance is ready.