Upgrading Kops Clusters

Upgrading kops

To upgrade the kops release, there are a couple of options. First check to see if there a new release of geodesic with an upgraded version of kops. If there is one and this is the desired release, update the FROM image tag in the geodesic module to point to this release.

To explictly upgrade kops, add this to the Dockerfile.

RUN make -C /packages/install kops KOPS_VERSION=1.9.0

Find the latest release of kops here.

Then follow the instructions below for upgrading Kubernetes.


Before attempting to upgrade any Kops cluster, ensure that it is in a healthy state by running kops validate cluster. If the cluster is not in a healthy state, do not proceed because the kops rolling-update process relies on the cluster state in order to determine if it should proceed. For example, if any Pods are in a crash loop, we recommend deleting or fixing those apps before attempting any upgrades.

Upgrading Kubernetes Release

Upgrading kops cluster to a new release of kubernetes requires that we upgrade the kubectl client in geodesic as well as select a kubernetes release that is compatible with kops. Note, kops usually lags behind in support for the latest release of Kubernetes. If the latest release of kops is v1.9, then that means it supports up to kubernetes v1.9 and will not be compatible with v1.10 or newer.

To explictly upgrade kubectl, add this to the Dockerfile

RUN make -C /packages/install kubectl KUBECTL_VERSION=1.10.0

The kubectl release version corresponds to the kubernetes release. Find the latest release of kubernetes here.

Then follow the official instructions for upgrading kops.

Set the KUBERNETES_VERSION in the Dockerfile

Then follow the instructions to update the manifest and update the cluster.

Upgrading EC2 Image for Nodes

Identify the AMI that should be used by following the official documentation. The latest AMI is published in the stable channel manifest.

The AMI will look something like this:


Set the KOPS_BASE_IMAGE to the latest release AMI in the Geodesic Module’s Dockerfile and rebuild the container.

Then rebuild the manifest and update the cluster

Update Manifest

The build-kops-manifest command will regenerate the /conf/kops/manifest.yaml file.


Login to AWS by running assume-role and following the prompts.


Update the cluster configuration.

kops replace -f /conf/kops/manifest.yaml

Apply the configuration changes. Note, this does not update any nodes.

kops update cluster --yes

Some changes will require rebuilding the nodes (for example, resizing nodes or changing the AMI).

To do this, you’ll need to run:

kops rolling-update cluster --yes

Read More