Using Geodesic with Kops

Learn how to provision kops clusters using Geodesic Modules.

Prerequisites

This assumes you’ve followed the Geodesic Module Usage with Terraform guide which covers all the scaffolding necessary to get started.

Geodesic uses kops to manage kubernetes clusters.

Create Cluster

Provisioning a kops cluster takes three steps:

  1. Provision a terraform-aws-kops-state-backend which consists of an S3 bucket, cluster DNS zone, and SSH keypair to access the k8s masters and nodes.
  2. Update the Dockerfile and rebuild the Geodesic Module to generate a kops manifest file (and restart shell)
  3. Launch a kops cluster from the manifest file

Provision State Backend

Configure Environment Variables

Update the environment variables in the module’s Dockerfile:

Example

ENV KOPS_CLUSTER_NAME=us-west-2.staging.example.com

ENV TF_VAR_kops_cluster_name=${KOPS_CLUSTER_NAME}
ENV TF_VAR_parent_zone_name=staging.example.com

Replace with values to suit your specific project. Note, the variables correspond to the outputs of the terraform-aws-kops-state-backend module, which follows a strict naming convention.

Rebuild Module

Rebuild the module

make docker/build

Add Kops State Terraform Module

Create a file in ./conf/aws-kops-backend/main.tf with following content

./conf/aws-kops-backend/main.tf

terraform {
  required_version = ">= 0.11.2"
  backend "s3" {}
}

variable "aws_assume_role_arn" {}

variable "tfstate_namespace" {}

variable "tfstate_stage" {}

variable "tfstate_region" {}

variable "kops_cluster_name" {}

variable "parent_zone_name" {}

provider "aws" {
  assume_role {
    role_arn = "${var.aws_assume_role_arn}"
  }
}

module "kops_state_backend" {
  source           = "git::https://github.com/cloudposse/terraform-aws-kops-state-backend.git?ref=tags/0.1.3"
  namespace        = "${var.tfstate_namespace}"
  stage            = "${var.tfstate_stage}"
  name             = "kops-state"
  parent_zone_name = "${var.parent_zone_name}"
  zone_name        = "$${name}.$${parent_zone_name}"
  cluster_name     = "${var.tfstate_region}"
  region           = "${var.tfstate_region}"
}

module "ssh_key_pair" {
  source              = "git::https://github.com/cloudposse/terraform-aws-key-pair.git?ref=tags/0.2.3"
  namespace           = "${var.tfstate_namespace}"
  stage               = "${var.tfstate_stage}"
  name                = "kops-${var.tfstate_region}"
  ssh_public_key_path = "/secrets/tf/ssh"
  generate_ssh_key    = "true"
}

output "parent_zone_id" {
  value = "${module.kops_state_backend.parent_zone_id}"
}

output "parent_zone_name" {
  value = "${module.kops_state_backend.parent_zone_name}"
}

output "zone_id" {
  value = "${module.kops_state_backend.zone_id}"
}

output "zone_name" {
  value = "${module.kops_state_backend.zone_name}"
}

output "bucket_name" {
  value = "${module.kops_state_backend.bucket_name}"
}

output "bucket_region" {
  value = "${module.kops_state_backend.bucket_region}"
}

output "bucket_domain_name" {
  value = "${module.kops_state_backend.bucket_domain_name}"
}

output "bucket_id" {
  value = "${module.kops_state_backend.bucket_id}"
}

output "bucket_arn" {
  value = "${module.kops_state_backend.bucket_arn}"
}

output "ssh_key_name" {
  value = "${module.ssh_key_pair.key_name}"
}

output "ssh_public_key" {
  value = "${module.ssh_key_pair.public_key}"
}

Start the Shell

Run the Geodesic shell. The wrapper script is installed in /usr/local/bin/$CLUSTER_NAME, so you should be able to just run something like:

$CLUSTER_NAME

Run the Geodesic Shell

staging.example.com
# Mounting /home/goruha into container
# Starting new staging.example.com session from cloudposse/staging.example.com:dev
# Exposing port 41179
* Started EC2 metadata service at http://169.254.169.254/latest

         _              _                                              _
     ___| |_ __ _  __ _(_)_ __   __ _    _____  ____ _ _ __ ___  _ __ | | ___
    / __| __/ _` |/ _` | | '_ \ / _` |  / _ \ \/ / _` | '_ ` _ \| '_ \| |/ _ \
    \__ \ || (_| | (_| | | | | | (_| | |  __/>  < (_| | | | | | | |_) | |  __/
    |___/\__\__,_|\__, |_|_| |_|\__, |  \___/_/\_\__,_|_| |_| |_| .__/|_|\___|
                  |___/         |___/                           |_|


IMPORTANT:
* Your $HOME directory has been mounted to `/localhost`
* Use `aws-vault` to manage your sessions
* Run `assume-role` to start a session


-> Run 'assume-role' to login to AWS
 ⧉  staging example
❌   (none) ~ ➤

Authorize on AWS

Assume role by running

assume-role

Assume role

❌   (none) conf ➤  assume-role
Enter passphrase to unlock /conf/.awsvault/keys/:
Enter token for arn:aws:iam::xxxxxxx:mfa/goruha: 781874
* Assumed role arn:aws:iam::xxxxxxx:role/OrganizationAccountAccessRole
-> Run 'init-terraform' to use this project
 ⧉  staging example
✅   (example-staging-admin) conf ➤

Provision aws-kops-backend

Change directory to /conf/aws-kops-backend and run there commands to provision the aws-kopstate-backend backend (S3 bucket, DNS zone, and SSH keypair)

init-terraform
terraform plan
terraform apply

From the Terraform outputs, copy the zone_name and bucket_name into the ENV vars KOPS_DNS_ZONE and KOPS_STATE_STORE in the Dockerfile.

terraform apply


In the example the bucket name is bucket_name = example-staging-kops-state and zone_name = us-west-2.staging.example.com. The public and private SSH keys are created and stored automatically in the encrypted S3 bucket.

Configure Environment Variables

Add to module’s Dockerfile the following environment variables

Example

# AWS Region of the S3 bucket to store cluster configuration
ENV KOPS_STATE_STORE=s3://example-staging-kops-state
ENV KOPS_STATE_STORE_REGION=us-west-2
ENV KOPS_DNS_ZONE=us-west-2.staging.example.com

## Config /etc/fstab to mount s3 bucket that containes generated ssh key
RUN s3 fstab '${TF_BUCKET}' '/' '/secrets/tf'

Replace with values to suit your specific project.

Rebuild Module

Rebuild the module

make docker/build

Configure Kops Manifest

Geodesic creates a kops cluster from a manifest. Kops manifest is yaml file that describe resources that determinates Kubernetes cluster. Geodesic generates the manifest from a template that supports gomplate interpolation for environment variables.

The manifest template (gomplate) is located in /templates/kops/default.yaml and is compiled to /conf/kops/manifest.yaml by running the build-kops-manifest script as a RUN step in the Dockerfile.

The geodesic module can overload the template if a different architecture is desired. All of our examples will rely on our default manifest.

Configure Environment Variables

Add to the module Dockerfile environment variables

Dockerfile

FROM cloudposse/geodesic:0.9.16

ENV BANNER staging example

### Default AWS Profile name
ENV AWS_DEFAULT_PROFILE="example-staging-admin"

ENV AWS_REGION="us-west-2"

ENV TF_VAR_tfstate_namespace="example"
ENV TF_VAR_tfstate_stage="staging"
ENV TF_VAR_tfstate_region="${AWS_REGION}"

ENV TF_BUCKET="example-staging-terraform-state"
ENV TF_BUCKET_REGION="${AWS_REGION}"
ENV TF_DYNAMODB_TABLE="example-staging-terraform-state-lock"

ENV KOPS_CLUSTER_NAME=us-west-2.staging.example.com

ENV TF_VAR_kops_cluster_name="${KOPS_CLUSTER_NAME}"
ENV TF_VAR_parent_zone_name="staging.example.com"

RUN s3 fstab '${TF_BUCKET}' '/' '/secrets/tf'

# AWS Region of the S3 bucket to store cluster configuration
ENV KOPS_STATE_STORE "s3://example-staging-kops-state"
ENV KOPS_STATE_STORE_REGION "${AWS_REGION}"

# https://github.com/kubernetes/kops/blob/master/channels/stable
# https://github.com/kubernetes/kops/blob/master/docs/images.md
ENV KOPS_BASE_IMAGE="kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08"
ENV KOPS_DNS_ZONE "${KOPS_CLUSTER_NAME}"
ENV KOPS_BASTION_PUBLIC_NAME="bastion"
ENV KOPS_PRIVATE_SUBNETS="172.20.32.0/19,172.20.64.0/19,172.20.96.0/19,172.20.128.0/19"
ENV KOPS_UTILITY_SUBNETS="172.20.0.0/22,172.20.4.0/22,172.20.8.0/22,172.20.12.0/22"

ENV KOPS_AVAILABILITY_ZONES="us-west-2a,us-west-2b,us-west-2c"
ENV KOPS_MASTERS_AVAILABILITY_ZONES="us-west-2a,us-west-2b,us-west-2c"
ENV KOPS_NODES_AVAILABILITY_ZONES="us-west-2b,us-west-2c"

# Instance sizes
ENV BASTION_MACHINE_TYPE "t2.micro"

# Kubernetes Master EC2 instance type (optional, required if the cluster uses Kubernetes)
ENV MASTER_MACHINE_TYPE "t2.small"

# Kubernetes Node EC2 instance type (optional, required if the cluster uses Kubernetes)
ENV NODE_MACHINE_TYPE "t2.medium"

# Kubernetes node count (Node EC2 instance count) (optional, required if the cluster uses Kubernetes)
ENV NODE_MIN_SIZE 4
ENV NODE_MAX_SIZE 4

RUN build-kops-manifest

You might want to adjust these settings:

  • BASTION_MACHINE_TYPE - EC2 instance type of bation node
  • MASTER_MACHINE_TYPE - EC2 instance type of masters
  • NODE_MACHINE_TYPE - EC2 instance type of EC2 worker nodes
  • NODE_MIN_SIZE - minimum number of worker nodes
  • NODE_MAX_SIZE - maximum number of worker nodes

Note, NODE_MIN_SIZE must be equal to or greater than the number of availability zones.

Rebuild Module

Rebuild the module

make docker/build

After building the manifest, we can apply it with kops to spin up or update the cluster.

Launch Cluster

Start Geodesic Shell

Run the Geodesic shell.

$CLUSTER_NAME
assume-role

Run the Geodesic Shell

staging.example.com
# Mounting /home/goruha into container
# Starting new staging.example.com session from cloudposse/staging.example.com:dev
# Exposing port 41179
* Started EC2 metadata service at http://169.254.169.254/latest

         _              _                                              _
     ___| |_ __ _  __ _(_)_ __   __ _    _____  ____ _ _ __ ___  _ __ | | ___
    / __| __/ _` |/ _` | | '_ \ / _` |  / _ \ \/ / _` | '_ ` _ \| '_ \| |/ _ \
    \__ \ || (_| | (_| | | | | | (_| | |  __/>  < (_| | | | | | | |_) | |  __/
    |___/\__\__,_|\__, |_|_| |_|\__, |  \___/_/\_\__,_|_| |_| |_| .__/|_|\___|
                  |___/         |___/                           |_|


IMPORTANT:
* Your $HOME directory has been mounted to `/localhost`
* Use `aws-vault` to manage your sessions
* Run `assume-role` to start a session


-> Run 'assume-role' to login to AWS
 ⧉  staging example
❌   (none) ~ ➤

Assume role

❌   (none) conf ➤  assume-role
Enter passphrase to unlock /conf/.awsvault/keys/:
Enter token for arn:aws:iam::xxxxxxx:mfa/goruha: 781874
* Assumed role arn:aws:iam::xxxxxxx:role/OrganizationAccountAccessRole
-> Run 'init-terraform' to use this project
 ⧉  staging example
✅   (example-staging-admin) conf ➤

Create Cluster

Run kops create -f /conf/kops/manifest.yaml to create the cluster (this will just create the cluster state and store it in the S3 bucket, but not the AWS resources for the cluster).

Example

✅   (example-staging-admin) kops ➤  kops create -f /conf/kops/manifest.yaml

Created cluster/us-west-2.staging.example.com
Created instancegroup/bastions
Created instancegroup/master-us-west-2a
Created instancegroup/master-us-west-2b
Created instancegroup/master-us-west-2c
Created instancegroup/nodes

To deploy these resources, run: kops update cluster us-west-2.staging.example.com --yes

 ⧉  staging example
✅   (example-staging-admin) kops ➤

Add SSH Keys

To add ssh keys generated previously, run the following command to mount the s3 bucket containing the SSH keys and register the SSH public key with the cluster.

Example

# Mount all S3 filesystems
mount -a

# Import SSH public key
kops create secret sshpublickey admin \
  -i /secrets/tf/ssh/example-staging-kops-us-west-2.pub \
  --name us-west-2.staging.example.com

Provision Cluster

Run the following to provision the AWS resources for the cluster. The --yes will apply the changes non-interactively.

kops update cluster --name us-west-2.staging.example.com --yes

kops update cluster --name us-west-2.staging.example.com --yes

✅   (example-staging-admin) ~ ➤  kops update cluster --name us-west-2.staging.example.com --yes
*********************************************************************************

A new kops version is available: 1.8.1

Upgrading is recommended
More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_kops.md#1.8.1

*********************************************************************************

W0517 12:16:53.870951     268 firewall.go:228] Opening etcd port on masters for access from the nodes, for calico.  This is unsafe in untrusted environments.
I0517 12:16:58.604951     268 executor.go:91] Tasks: 0 done / 131 total; 40 can run
I0517 12:17:01.145989     268 vfs_castore.go:430] Issuing new certificate: "ca"
I0517 12:17:01.162586     268 vfs_castore.go:430] Issuing new certificate: "apiserver-aggregator-ca"
I0517 12:17:05.839088     268 executor.go:91] Tasks: 40 done / 131 total; 38 can run
I0517 12:17:08.095988     268 vfs_castore.go:430] Issuing new certificate: "kube-scheduler"
I0517 12:17:08.216884     268 vfs_castore.go:430] Issuing new certificate: "kube-controller-manager"
I0517 12:17:08.300522     268 vfs_castore.go:430] Issuing new certificate: "kubelet"
I0517 12:17:08.382453     268 vfs_castore.go:430] Issuing new certificate: "apiserver-aggregator"
I0517 12:17:08.385249     268 vfs_castore.go:430] Issuing new certificate: "master"
I0517 12:17:08.489622     268 vfs_castore.go:430] Issuing new certificate: "kube-proxy"
I0517 12:17:08.513227     268 vfs_castore.go:430] Issuing new certificate: "apiserver-proxy-client"
I0517 12:17:08.578911     268 vfs_castore.go:430] Issuing new certificate: "kubecfg"
I0517 12:17:08.688453     268 vfs_castore.go:430] Issuing new certificate: "kops"
I0517 12:17:08.754063     268 vfs_castore.go:430] Issuing new certificate: "kubelet-api"
I0517 12:17:12.413322     268 executor.go:91] Tasks: 78 done / 131 total; 36 can run
I0517 12:17:17.010674     268 executor.go:91] Tasks: 114 done / 131 total; 10 can run
I0517 12:17:18.088422     268 logging_retryer.go:60] Retryable error (PriorRequestNotComplete: The request was rejected because Route 53 was still processing a prior request.
        status code: 400, request id: 3dcff81f-59cc-11e8-b19f-0b847b609fa5) from route53/ChangeResourceRecordSets - will retry after delay of 702ms
I0517 12:17:19.058746     268 executor.go:91] Tasks: 124 done / 131 total; 7 can run
I0517 12:17:19.355382     268 natgateway.go:269] Waiting for NAT Gateway "nat-0cf6ad5e93b87e3d2" to be available (this often takes about 5 minutes)
I0517 12:17:19.357305     268 natgateway.go:269] Waiting for NAT Gateway "nat-0b546e9de5a1d1b06" to be available (this often takes about 5 minutes)
I0517 12:17:19.647656     268 natgateway.go:269] Waiting for NAT Gateway "nat-028a04891ae90c4ce" to be available (this often takes about 5 minutes)
I0517 12:18:58.100048     268 executor.go:91] Tasks: 131 done / 131 total; 0 can run
I0517 12:18:58.100095     268 dns.go:153] Pre-creating DNS records
I0517 12:19:01.044909     268 update_cluster.go:248] Exporting kubecfg for cluster
kops has set your kubectl context to us-west-2.staging.example.com

Cluster is starting.  It should be ready in a few minutes.

Suggestions:
 * validate cluster: kops validate cluster
 * list nodes: kubectl get nodes --show-labels
 * ssh to the bastion: ssh -A -i ~/.ssh/id_rsa [email protected]
The admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
 * read about installing addons: https://github.com/kubernetes/kops/blob/master/docs/addons.md

All done. At this point, the kops cluster is now up and running (though it might take 5-10 minutes before all nodes come online).

Update Cluster

Run kops replace -f /conf/kops/manifest.yaml to update the cluster. This will just update the cluster state in the S3 bucket, but not modify any of the underlying AWS resources for the cluster.

Apply the Updates

Run the following command to update the AWS resources for the cluster. The --yes will apply the changes non-interactively.

kops update cluster --name us-west-2.staging.example.com --yes

All done. At this point, the kops cluster is now updated and running.

Export kubecfg

When you run into the Geodesic module shell you need to export the kubecfg which provides the TLS client certificates necessary for kubectl to authenticate with the cluster.

Run kops export kubecfg to export the kubecfg. This will export the kubecfg to the location set in the KUBECONFIG environment variable.

(Note, in older versions of kops you will need to pass the cluster name, so run kops export kubecfg $KOPS_CLUSTER_NAME)

✅   (example-staging-admin) ~ ➤  kops export kubecfg
kops has set your kubectl context to us-west-2.staging.example.com

By default, geodesic exports KUBECONFIG=/dev/shm to ensure this config does not pesist.

Provision Platform Backing Services

We provide a number of well-tested Terraform Modules to provision essential AWS resources needed by Helm Charts like external-dns and chart-repo. See our Terraform modules for Kubernetes (Kops).