How do we rotate the master SSH keys on a Kubernetes cluster provisioned with kops?

Learn how to rotate ssh keys on a kops cluster by generating new ones and then performing a rolling-update on the cluster to apply the changes.

Question

How do we rotate the master SSH keys on a Kubernetes cluster provisioned with kops?

Answer

We provision SSH keys in geodesic using terraform-aws-key-pair module. Following this documentation you need to taint ssh key with terraform, then provision new one and update the cluster.

To rotate SSH keys, follow these instructions:

Start the Geodesic Shell

Run the Geodesic shell followed by assume-role

$CLUSTER_NAME

Run the Geodesic Shell

staging.example.com
# Mounting /home/goruha into container
# Starting new staging.example.com session from cloudposse/staging.example.com:dev
# Exposing port 41179
* Started EC2 metadata service at http://169.254.169.254/latest

         _              _                                              _
     ___| |_ __ _  __ _(_)_ __   __ _    _____  ____ _ _ __ ___  _ __ | | ___
    / __| __/ _` |/ _` | | '_ \ / _` |  / _ \ \/ / _` | '_ ` _ \| '_ \| |/ _ \
    \__ \ || (_| | (_| | | | | | (_| | |  __/>  < (_| | | | | | | |_) | |  __/
    |___/\__\__,_|\__, |_|_| |_|\__, |  \___/_/\_\__,_|_| |_| |_| .__/|_|\___|
                  |___/         |___/                           |_|


IMPORTANT:
* Your $HOME directory has been mounted to `/localhost`
* Use `aws-vault` to manage your sessions
* Run `assume-role` to start a session


-> Run 'assume-role' to login to AWS
 ⧉  staging example
❌   (none) ~ ➤

Then login to AWS by running assume-role:

Assume role

❌   (none) conf ➤  assume-role
Enter passphrase to unlock /conf/.awsvault/keys/:
Enter token for arn:aws:iam::xxxxxxx:mfa/goruha: 781874
* Assumed role arn:aws:iam::xxxxxxx:role/OrganizationAccountAccessRole
-> Run 'init-terraform' to use this project
 ⧉  staging example
✅   (example-staging-admin) conf ➤

Configure kubectl and helm

When you start the Geodesic shell, you will need to export the kubecfg, which provides the TLS client certificates necessary for kubectl and helm to authenticate with the cluster.

Export kops config

✅   (example-staging-admin) ~ ➤  kops export kubecfg
kops has set your kubectl context to us-west-2.staging.example.com

(Note, in older versions of kops you will need to pass the cluster name, so run kops export kubecfg $KOPS_CLUSTER_NAME)

Recreate SSH Key

cd /conf/kops
init-terraform
terraform taint --module ssh_key_pair tls_private_key.default
terraform apply

Update Kubernetes Cluster

According to the instrunctions, execute the following commands to apply the new SSH key to the Kubernetes cluster:

s3 mount
kops delete secret --name us-west-2.staging.example.com \
  sshpublickey admin
kops create secret sshpublickey admin \
  -i /secrets/tf/ssh/example-staging-kops-us-west-2.pub \
  --name us-west-2.staging.example.com
kops update cluster --yes
kops rolling-update cluster --yes