Skip to main content
Sponsor @cloudposse

Philips Labs GitHub Action Runners

If we are not deploying EKS, it's not worth the additional effort to set up Self-Hosted runners on EKS. Instead, we deploy Self-Hosted runners on EC2 instances. These are managed by an API Gateway and Lambda function that will automatically scale the number of runners based on the number of pending jobs in the queue. The queue is written to by the API Gateway from GitHub Events.

Quick Start​

StepsActions
1. Create GitHub AppClickOps
2. Upload GitHub App ID and Private Key to AWS SSMSet SSM Param "/pl-github-runners/id" and "/pl-github-runners/key" (base64 encoded)
3. Deploy GitHub OIDC ProviderDeploy GitHub OIDC to every needed account
4. Deploy GitHub Runnersatmos terraform deploy philips-labs-github-runners -s core-use1-auto
5. Update Webhook (if changed or redeployed)ClickOps

Deploy​

The setup for the Philips Labs GitHub Action Runners requires first creating the GitHub App, then deploying the philips-labs-github-runner component, and then finalizing the GitHub App webhook. Cloud Posse typically does not have access to the customer's GitHub Organization settings, so the customer will need to create the initial GitHub App, then hand the setup back to Cloud Posse. Then Cloud Posse can deploy the component and generate the webhook. Finally, the customer will then need to add the webhook to the GitHub App and ensure the App is installed to all relevant GitHub repositories.

Follow the guide with the upstream module, philips-labs/terraform-aws-github-runner, or follow the steps below.

1 Vendor Components​

Vendor in the necessary components with the following workflow:

These are the commands included in the vendor/philips-labs workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Atmos workflow! 🚀

2 Create the GitHub App​

Customer Requirement

This step requires access to the GitHub Organization. Customers will need to create this GitHub App in Jumpstart engagements.

  1. Create a new GitHub App

  2. Choose a name

  3. Choose a website (mandatory, not required for the module).

  4. Disable the webhook for now (we will configure this later or create an alternative webhook).

  5. Add the following permission for your chosen runner scope:

    Repository Permissions​

    • Actions: Read-only (check for queued jobs)
    • Checks: Read-only (receive events for new builds)
    • Metadata: Read-only (default/required)
    • Administration: Read & write (to register runner)

  6. Generate a Private Key

  7. If you are working with Cloud Posse, upload this Private Key and GitHub App ID to 1Password and inform Cloud Posse. Otherwise, continue to the next step.

3 Upload AWS SSM Parameters​

tip

This step does not require access to the GitHub Organization. Cloud Posse will run this deployment for Jumpstart engagements.

Now that the GitHub App has been created, upload the Private Key and GitHub App ID to AWS SSM Parameter Store in core-use1-auto (or your chosen region).

  1. Upload the PEM file key to the specified ssm path, /pl-github-runners/key, in core-use1-auto as a base64 encoded string.
  2. Upload the GitHub App ID to the specified ssm path, /pl-github-runners/id, in core-use1-auto.

Or run the upload/pl-secrets workflow with atmos to write the GitHub App information to the core-use1-auto SSM account and deploy the component.

These are the commands included in the upload/pl-secrets workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Atmos workflow! 🚀

4 Deploy GitHub OIDC Providers​

First deploy the GitHub OIDC provider to all accounts where we want to grant GitHub access. The typical list of accounts is included with the deploy/github-oidc-provider workflow; run the following with SuperAdmin:

These are the commands included in the deploy/github-oidc-provider workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Atmos workflow! 🚀

5 Deploy the Philips Labs GitHub Runners​

Now that the GitHub App has been created and the SSM parameters have been uploaded, deploy the philips-labs-github-runners component.

These are the commands included in the deploy/pl-github-runners workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Atmos workflow! 🚀

6 Add the Webhook to the GitHub App​

Customer Requirement

This step requires access to the GitHub Organization. Customers will need to finalize the GitHub App in Jumpstart engagements.

Now that the component is deployed and the webhook has been created, add that webhook to the GitHub App. Both the webhook URL and secret should now be stored in 1Password. If not, you can retrieve these values from the output of the philips-labs-github-runners component in core-use1-auto as described in the previous step.

  1. Open the GitHub App created in Create the GitHub App above
  2. Enable the webhook.
  3. Provide the webhook url, should be part of the output of terraform.
  4. Provide the webhook secret (terraform output -raw <NAME_OUTPUT_VAR>).
  5. In the "Permissions & Events" section and then "Subscribe to Events" subsection, check "Workflow Job".
  6. Ensure the webhook for the GitHub app is enabled and pointing to the output of the module. The endpoint can be found by running atmos terraform output philips-labs-github-runners -s core-use1-auto 'webhook'

Usage​

Once you've deployed Self Hosted runners select the appropriate runner set with the runs-on configuration in any GitHub Actions workflow. For example, we can use the default runner set as such:

runs-on: ["self-hosted", "default"]

However, it's very likely you will have resource-intensive jobs that the default runner size may not satisfy. We recommend deploying additional runner sets for each tier of workflow resource requirements. For example in our internal GitHub Organization, we have default, medium, and large runners.

Using the terraform Label​

By default, we configure the Atmos Terraform GitHub Actions to use the terraform labeled Self-Hosted runners.

runs-on: ["self-hosted", "terraform"]

However also by default we only have the single runner set. We recommend deploying a second runner set with a larger resource allocation for these specific jobs.

Remove the terraform label from the default runner set and add the terraform label to your new, larger runner set. Since the workflows are all labeled with terraform already, they will automatically select the new runner set on their next run.

FAQ​

I cannot assume the role from GitHub Actions after deploying​

The following error is very common if the GitHub workflow is missing proper permission.

Error: User: arn:aws:sts::***:assumed-role/acme-core-use1-auto-actions-runner@actions-runner-system/token-file-web-identity is not authorized to perform: sts:TagSession on resource: arn:aws:iam::999999999999:role/acme-plat-use1-dev-gha

In order to use a web identity, GitHub Action pipelines must have the following permission. See GitHub Action documentation for more.

permissions:
  id-token: write # This is required for requesting the JWT
  contents: read # This is required for actions/checkout