Skip to main content

Component: eks/cluster

This component is responsible for provisioning an end-to-end EKS Cluster, including managed node groups and Fargate profiles.

Windows not supported

This component has not been tested with Windows worker nodes of any launch type. Although upstream modules support Windows nodes, there are likely issues around incorrect or insufficient IAM permissions or other configuration that would need to be resolved for this component to properly configure the upstream modules for Windows nodes. If you need Windows nodes, please experiment and be on the lookout for issues, and then report any issues to Cloud Posse.

Usage​

Stack Level: Regional

Here's an example snippet for how to use this component.

This example expects the Cloud Posse Reference Architecture Identity and Network designs deployed for mapping users to EKS service roles and granting access in a private network. In addition, this example has the GitHub OIDC integration added and makes use of Karpenter to dynamically scale cluster nodes.

For more on these requirements, see Identity Reference Architecture, Network Reference Architecture, the GitHub OIDC component, and the Karpenter component.

Mixin pattern for Kubernetes version​

We recommend separating out the Kubernetes and related addons versions into a separate mixin (one per Kubernetes minor version), to make it easier to run different versions in different environments, for example while testing a new version.

We also recommend leaving "resolve conflicts" settings unset and therefore using the default "OVERWRITE" setting because any custom configuration that you would want to preserve should be managed by Terraform configuring the add-ons directly.

For example, create catalog/eks/cluster/mixins/k8s-1-29.yaml with the following content:

components:
  terraform:
    eks/cluster:
      vars:
        cluster_kubernetes_version: "1.29"

        # You can set all the add-on versions to `null` to use the latest version,
        # but that introduces drift as new versions are released. As usual, we recommend
        # pinning the versions to a specific version and upgrading when convenient.

        # Determine the latest version of the EKS add-ons for the specified Kubernetes version
        #  EKS_K8S_VERSION=1.29 # replace with your cluster version
        #  ADD_ON=vpc-cni # replace with the add-on name
        #  echo "${ADD_ON}:" && aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION --addon-name $ADD_ON \
        #  --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

        # To see versions for all the add-ons, wrap the above command in a for loop:
        #   for ADD_ON in vpc-cni kube-proxy coredns aws-ebs-csi-driver aws-efs-csi-driver; do
        #     echo "${ADD_ON}:" && aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION --addon-name $ADD_ON \
        #     --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table
        #   done

        # To see the custom configuration schema for an add-on, run the following command:
        #   aws eks describe-addon-configuration --addon-name aws-ebs-csi-driver \
        #   --addon-version v1.20.0-eksbuild.1 | jq '.configurationSchema | fromjson'
        # See the `coredns` configuration below for an example of how to set a custom configuration.

        # https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html
        # https://docs.aws.amazon.com/eks/latest/userguide/managing-add-ons.html#creating-an-add-on
        addons:
          # https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
          # https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html#cni-iam-role-create-role
          # https://aws.github.io/aws-eks-best-practices/networking/vpc-cni/#deploy-vpc-cni-managed-add-on
          vpc-cni:
            addon_version: "v1.16.0-eksbuild.1" # set `addon_version` to `null` to use the latest version
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html
          kube-proxy:
            addon_version: "v1.29.0-eksbuild.1" # set `addon_version` to `null` to use the latest version
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html
          coredns:
            addon_version: "v1.11.1-eksbuild.4" # set `addon_version` to `null` to use the latest version
            ## override default replica count of 2. In very large clusters, you may want to increase this.
            configuration_values: '{"replicaCount": 3}'

          # https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html
          # https://aws.amazon.com/blogs/containers/amazon-ebs-csi-driver-is-now-generally-available-in-amazon-eks-add-ons
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html#csi-iam-role
          # https://github.com/kubernetes-sigs/aws-ebs-csi-driver
          aws-ebs-csi-driver:
            addon_version: "v1.27.0-eksbuild.1" # set `addon_version` to `null` to use the latest version
            # If you are not using [volume snapshots](https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/#how-to-use-volume-snapshots)
            # (and you probably are not), disable the EBS Snapshotter
            # See https://github.com/aws/containers-roadmap/issues/1919
            configuration_values: '{"sidecars":{"snapshotter":{"forceEnable":false}}}'

          aws-efs-csi-driver:
            addon_version: "v1.7.7-eksbuild.1" # set `addon_version` to `null` to use the latest version
            # Set a short timeout in case of conflict with an existing efs-controller deployment
            create_timeout: "7m"

Common settings for all Kubernetes versions​

In your main stack configuration, you can then set the Kubernetes version by importing the appropriate mixin:

#
import:
  - catalog/eks/cluster/mixins/k8s-1-29

components:
  terraform:
    eks/cluster:
      vars:
        enabled: true
        name: eks
        vpc_component_name: "vpc"
        eks_component_name: "eks/cluster"

        # Your choice of availability zones or availability zone ids
        # availability_zones: ["us-east-1a", "us-east-1b", "us-east-1c"]
        aws_ssm_agent_enabled: true
        allow_ingress_from_vpc_accounts:
          - tenant: core
            stage: auto
          - tenant: core
            stage: corp
          - tenant: core
            stage: network

        public_access_cidrs: []
        allowed_cidr_blocks: []
        allowed_security_groups: []

        enabled_cluster_log_types:
          # Caution: enabling `api` log events may lead to a substantial increase in Cloudwatch Logs expenses.
          - api
          - audit
          - authenticator
          - controllerManager
          - scheduler

        oidc_provider_enabled: true

        # Allows GitHub OIDC role
        github_actions_iam_role_enabled: true
        github_actions_iam_role_attributes: ["eks"]
        github_actions_allowed_repos:
          - acme/infra

        # We recommend, at a minimum, deploying 1 managed node group,
        # with the same number of instances as availability zones (typically 3).
        managed_node_groups_enabled: true
        node_groups: # for most attributes, setting null here means use setting from node_group_defaults
          main:
            # availability_zones = null will create one autoscaling group
            # in every private subnet in the VPC
            availability_zones: null

            # Tune the desired and minimum group size according to your baseload requirements.
            # We recommend no autoscaling for the main node group, so it will
            # stay at the specified desired group size, with additional
            # capacity provided by Karpenter. Nevertheless, we recommend
            # deploying enough capacity in the node group to handle your
            # baseload requirements, and in production, we recommend you
            # have a large enough node group to handle 3/2 (1.5) times your
            # baseload requirements, to handle the loss of a single AZ.
            desired_group_size: 3 # number of instances to start with, should be >= number of AZs
            min_group_size: 3 # must be  >= number of AZs
            max_group_size: 3

            # Can only set one of ami_release_version or kubernetes_version
            # Leave both null to use latest AMI for Cluster Kubernetes version
            kubernetes_version: null # use cluster Kubernetes version
            ami_release_version: null # use latest AMI for Kubernetes version

            attributes: []
            create_before_destroy: true
            cluster_autoscaler_enabled: true
            instance_types:
              # Tune the instance type according to your baseload requirements.
              - c7a.medium
            ami_type: AL2_x86_64 # use "AL2_x86_64" for standard instances, "AL2_x86_64_GPU" for GPU instances
            node_userdata:
              # WARNING: node_userdata is alpha status and will likely change in the future.
              #          Also, it is only supported for AL2 and some Windows AMIs, not BottleRocket or AL2023.
              # Kubernetes docs: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
              kubelet_extra_args: >-
                --kube-reserved cpu=100m,memory=0.6Gi,ephemeral-storage=1Gi
                --system-reserved cpu=100m,memory=0.2Gi,ephemeral-storage=1Gi
                --eviction-hard memory.available<200Mi,nodefs.available<10%,imagefs.available<15%
            block_device_map:
              # EBS volume for local ephemeral storage
              # IGNORED if legacy `disk_encryption_enabled` or `disk_size` are set!
              # Use "/dev/xvda" for most of the instances (without local NVMe)
              # using most of the Linuxes, "/dev/xvdb" for BottleRocket
              "/dev/xvda":
                ebs:
                  volume_size: 100 # number of GB
                  volume_type: gp3

            kubernetes_labels: {}
            kubernetes_taints: {}
            resources_to_tag:
              - instance
              - volume
            tags: null

        # The abbreviation method used for Availability Zones in your project.
        # Used for naming resources in managed node groups.
        # Either "short" or "fixed".
        availability_zone_abbreviation_type: fixed

        cluster_private_subnets_only: true
        cluster_encryption_config_enabled: true
        cluster_endpoint_private_access: true
        cluster_endpoint_public_access: false
        cluster_log_retention_period: 90

        # List of `aws-team-roles` (in the account where the EKS cluster is deployed) to map to Kubernetes RBAC groups
        # You cannot set `system:*` groups here, except for `system:masters`.
        # The `idp:*` roles referenced here are created by the `eks/idp-roles` component.
        # While set here, the `idp:*` roles will have no effect until after
        # the `eks/idp-roles` component is applied, which must be after the
        # `eks/cluster` component is deployed.
        aws_team_roles_rbac:
          - aws_team_role: admin
            groups:
              - system:masters
          - aws_team_role: poweruser
            groups:
              - idp:poweruser
          - aws_team_role: observer
            groups:
              - idp:observer
          - aws_team_role: planner
            groups:
              - idp:observer
          - aws_team: terraform
            groups:
              - system:masters

        # Permission sets from AWS SSO allowing cluster access
        # See `aws-sso` component.
        aws_sso_permission_sets_rbac:
          - aws_sso_permission_set: PowerUserAccess
            groups:
              - idp:poweruser

        # Set to false if you are not using Karpenter
        karpenter_iam_role_enabled: true

        # All Fargate Profiles will use the same IAM Role when `legacy_fargate_1_role_per_profile_enabled` is set to false.
        # Recommended for all new clusters, but will damage existing clusters provisioned with the legacy component.
        legacy_fargate_1_role_per_profile_enabled: false
        # While it is possible to deploy add-ons to Fargate Profiles, it is not recommended. Use a managed node group instead.
        deploy_addons_to_fargate: false

Amazon EKS End-of-Life Dates​

When picking a Kubernetes version, be sure to review the end-of-life dates for Amazon EKS. Refer to the chart below:

cyclereleaselatestlatest releaseeolextended support
1.292024-01-231.29-eks-62024-04-182025-03-232026-03-23
1.282023-09-261.28-eks-122024-04-182024-11-262025-11-26
1.272023-05-241.27-eks-162024-04-182024-07-242025-07-24
1.262023-04-111.26-eks-172024-04-182024-06-112025-06-11
1.252023-02-211.25-eks-182024-04-182024-05-012025-05-01
1.242022-11-151.24-eks-212024-04-182024-01-312025-01-31
1.232022-08-111.23-eks-232024-04-182023-10-112024-10-11
1.222022-04-041.22-eks-142023-06-302023-06-042024-09-01
1.212021-07-191.21-eks-182023-06-092023-02-162024-07-15
1.202021-05-181.20-eks-142023-05-052022-11-01False
1.192021-02-161.19-eks-112022-08-152022-08-01False
1.182020-10-131.18-eks-132022-08-152022-08-15False

* This Chart was generated 2024-05-12 with the eol tool. Install it with python3 -m pip install --upgrade norwegianblue and create a new table by running eol --md amazon-eks locally, or view the information by visiting the endoflife website.

You can also view the release and support timeline for the Kubernetes project itself.

Using Addons​

EKS clusters support “Addons” that can be automatically installed on a cluster. Install these addons with the var.addons input.

info

Run the following command to see all available addons, their type, and their publisher. You can also see the URL for addons that are available through the AWS Marketplace. Replace 1.27 with the version of your cluster. See Creating an addon for more details.

EKS_K8S_VERSION=1.29 # replace with your cluster version
aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION \
  --query 'addons[].{MarketplaceProductUrl: marketplaceInformation.productUrl, Name: addonName, Owner: owner Publisher: publisher, Type: type}' --output table
info

You can see which versions are available for each addon by executing the following commands. Replace 1.29 with the version of your cluster.

EKS_K8S_VERSION=1.29 # replace with your cluster version
echo "vpc-cni:" && aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION --addon-name vpc-cni \
  --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

echo "kube-proxy:" && aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION --addon-name kube-proxy \
  --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

echo "coredns:" && aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION --addon-name coredns \
  --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

echo "aws-ebs-csi-driver:" && aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION --addon-name aws-ebs-csi-driver \
  --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

echo "aws-efs-csi-driver:" && aws eks describe-addon-versions --kubernetes-version $EKS_K8S_VERSION --addon-name aws-efs-csi-driver \
  --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

Some add-ons accept additional configuration. For example, the vpc-cni addon accepts a disableNetworking parameter. View the available configuration options (as JSON Schema) via the aws eks describe-addon-configuration command. For example:

aws eks describe-addon-configuration \
  --addon-name aws-ebs-csi-driver \
  --addon-version v1.20.0-eksbuild.1 | jq '.configurationSchema | fromjson'

You can then configure the add-on via the configuration_values input. For example:

aws-ebs-csi-driver:
  configuration_values: '{"node": {"loggingFormat": "json"}}'

Configure the addons like the following example:

# https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html
# https://docs.aws.amazon.com/eks/latest/userguide/managing-add-ons.html#creating-an-add-on
# https://aws.amazon.com/blogs/containers/amazon-eks-add-ons-advanced-configuration/
addons:
  # https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html
  # https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
  # https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html#cni-iam-role-create-role
  # https://aws.github.io/aws-eks-best-practices/networking/vpc-cni/#deploy-vpc-cni-managed-add-on
  vpc-cni:
    addon_version: "v1.12.2-eksbuild.1" # set `addon_version` to `null` to use the latest version
  # https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html
  kube-proxy:
    addon_version: "v1.25.6-eksbuild.1" # set `addon_version` to `null` to use the latest version
  # https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html
  coredns:
    addon_version: "v1.9.3-eksbuild.2" # set `addon_version` to `null` to use the latest version
    # Override default replica count of 2, to have one in each AZ
    configuration_values: '{"replicaCount": 3}'
  # https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html
  # https://aws.amazon.com/blogs/containers/amazon-ebs-csi-driver-is-now-generally-available-in-amazon-eks-add-ons
  # https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html#csi-iam-role
  # https://github.com/kubernetes-sigs/aws-ebs-csi-driver
  aws-ebs-csi-driver:
    addon_version: "v1.19.0-eksbuild.2" # set `addon_version` to `null` to use the latest version
    # If you are not using [volume snapshots](https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/#how-to-use-volume-snapshots)
    # (and you probably are not), disable the EBS Snapshotter with:
    configuration_values: '{"sidecars":{"snapshotter":{"forceEnable":false}}}'

Some addons, such as CoreDNS, require at least one node to be fully provisioned first. See issue #170 for more details. Set var.addons_depends_on to true to require the Node Groups to be provisioned before addons.

addons_depends_on: true
addons:
  coredns:
    addon_version: "v1.8.7-eksbuild.1"
danger

Addons may not be suitable for all use-cases! For example, if you are deploying Karpenter to Fargate and using Karpenter to provision all nodes, these nodes will never be available before the cluster component is deployed if you are using the CoreDNS addon (for example).

This is one of the reasons we recommend deploying a managed node group: to ensure that the addons will become fully functional during deployment of the cluster.

For more information on upgrading EKS Addons, see "How to Upgrade EKS Cluster Addons"

Adding and Configuring a new EKS Addon​

The component already supports all the EKS addons shown in the configurations above. To add a new EKS addon, not supported by the cluster, add it to the addons map (addons variable):

addons:
  my-addon:
    addon_version: "..."

If the new addon requires an EKS IAM Role for Kubernetes Service Account, perform the following steps:

  • Add a file addons-custom.tf to the eks/cluster folder if not already present

  • In the file, add an IAM policy document with the permissions required for the addon, and use the eks-iam-role module to provision an IAM Role for Kubernetes Service Account for the addon:

      data "aws_iam_policy_document" "my_addon" {
        statement {
          sid       = "..."
          effect    = "Allow"
          resources = ["..."]

          actions = [
            "...",
            "..."
          ]
        }
      }

      module "my_addon_eks_iam_role" {
        source  = "cloudposse/eks-iam-role/aws"
        version = "2.1.0"

        eks_cluster_oidc_issuer_url = local.eks_cluster_oidc_issuer_url

        service_account_name      = "..."
        service_account_namespace = "..."

        aws_iam_policy_document = [one(data.aws_iam_policy_document.my_addon[*].json)]

        context = module.this.context
      }

    For examples of how to configure the IAM role and IAM permissions for EKS addons, see addons.tf.

  • Add a file additional-addon-support_override.tf to the eks/cluster folder if not already present

  • In the file, add the IAM Role for Kubernetes Service Account for the addon to the overridable_additional_addon_service_account_role_arn_map map:

      locals {
        overridable_additional_addon_service_account_role_arn_map = {
          my-addon = module.my_addon_eks_iam_role.service_account_role_arn
        }
      }

  • This map will override the default map in the additional-addon-support.tf file, and will be merged into the final map together with the default EKS addons vpc-cni and aws-ebs-csi-driver (which this component configures and creates IAM Roles for Kubernetes Service Accounts)

  • Follow the instructions in the additional-addon-support.tf file if the addon may need to be deployed to Fargate, or has dependencies that Terraform cannot detect automatically.

Requirements​

NameVersion
terraform>= 1.3.0
aws>= 4.9.0
random>= 3.0

Providers​

NameVersion
aws>= 4.9.0
random>= 3.0

Modules​

NameSourceVersion
aws_ebs_csi_driver_eks_iam_rolecloudposse/eks-iam-role/aws2.1.1
aws_ebs_csi_driver_fargate_profilecloudposse/eks-fargate-profile/aws1.3.0
aws_efs_csi_driver_eks_iam_rolecloudposse/eks-iam-role/aws2.1.1
coredns_fargate_profilecloudposse/eks-fargate-profile/aws1.3.0
eks_clustercloudposse/eks-cluster/aws4.1.0
fargate_pod_execution_rolecloudposse/eks-fargate-profile/aws1.3.0
fargate_profilecloudposse/eks-fargate-profile/aws1.3.0
iam_arns../../account-map/modules/roles-to-principalsn/a
iam_roles../../account-map/modules/iam-rolesn/a
karpenter_labelcloudposse/label/null0.25.0
region_node_group./modules/node_group_by_regionn/a
thiscloudposse/label/null0.25.0
utilscloudposse/utils/aws1.3.0
vpccloudposse/stack-config/yaml//modules/remote-state1.5.0
vpc_cni_eks_iam_rolecloudposse/eks-iam-role/aws2.1.1
vpc_ingresscloudposse/stack-config/yaml//modules/remote-state1.5.0

Resources​

NameType
aws_iam_instance_profile.defaultresource
aws_iam_policy.ipv6_eks_cni_policyresource
aws_iam_role.karpenterresource
aws_iam_role_policy_attachment.amazon_ec2_container_registry_readonlyresource
aws_iam_role_policy_attachment.amazon_eks_worker_node_policyresource
aws_iam_role_policy_attachment.amazon_ssm_managed_instance_coreresource
aws_iam_role_policy_attachment.aws_ebs_csi_driverresource
aws_iam_role_policy_attachment.aws_efs_csi_driverresource
aws_iam_role_policy_attachment.ipv6_eks_cni_policyresource
aws_iam_role_policy_attachment.vpc_cniresource
random_pet.camel_case_warningresource
aws_availability_zones.defaultdata source
aws_iam_policy_document.assume_roledata source
aws_iam_policy_document.ipv6_eks_cni_policydata source
aws_iam_policy_document.vpc_cni_ipv6data source
aws_iam_roles.sso_rolesdata source
aws_partition.currentdata source

Inputs​

NameDescriptionTypeDefaultRequired
access_configAccess configuration for the EKS cluster
object({
    authentication_mode                         = optional(string, "API")
    bootstrap_cluster_creator_admin_permissions = optional(bool, false)
  })
{}no
additional_tag_mapAdditional key-value pairs to add to each map in tags_as_list_of_maps. Not added to tags or id.
This is for some rare cases where resources want additional configuration of tags
and therefore take a list of maps with tag key, value, and additional configuration.		map(string){}no
addonsManages EKS addons resources
map(object({
    enabled       = optional(bool, true)
    addon_version = optional(string, null)
    # configuration_values is a JSON string, such as '{"computeType": "Fargate"}'.
    configuration_values = optional(string, null)
    # Set default resolve_conflicts to OVERWRITE because it is required on initial installation of
    # add-ons that have self-managed versions installed by default (e.g. vpc-cni, coredns), and
    # because any custom configuration that you would want to preserve should be managed by Terraform.
    resolve_conflicts_on_create = optional(string, "OVERWRITE")
    resolve_conflicts_on_update = optional(string, "OVERWRITE")
    service_account_role_arn    = optional(string, null)
    create_timeout              = optional(string, null)
    update_timeout              = optional(string, null)
    delete_timeout              = optional(string, null)
  }))
{}no
addons_depends_onIf set true (recommended), all addons will depend on managed node groups provisioned by this component and therefore not be installed until nodes are provisioned.
See issue #170 for more details.		booltrueno
allow_ingress_from_vpc_accountsList of account contexts to pull VPC ingress CIDR and add to cluster security group.

e.g.

{
environment = "ue2",
stage = "auto",
tenant = "core"
}		any[]no
allowed_cidr_blocksList of CIDR blocks to be allowed to connect to the EKS clusterlist(string)[]no
allowed_security_groupsList of Security Group IDs to be allowed to connect to the EKS clusterlist(string)[]no
apply_config_map_aws_auth(Obsolete) Whether to execute kubectl apply to apply the ConfigMap to allow worker nodes to join the EKS cluster.
This input is included to avoid breaking existing configurations that set it to true;
a value of false is no longer allowed.
This input is obsolete and will be removed in a future release.		booltrueno
attributesID element. Additional attributes (e.g. workers or cluster) to add to id,
in the order they appear in the list. New attributes are appended to the
end of the list. The elements of the list are joined by the delimiter
and treated as a single ID element.		list(string)[]no
availability_zone_abbreviation_typeType of Availability Zone abbreviation (either fixed or short) to use in names. See https://github.com/cloudposse/terraform-aws-utils for details.string"fixed"no
availability_zone_idsList of Availability Zones IDs where subnets will be created. Overrides availability_zones.
Can be the full name, e.g. use1-az1, or just the part after the AZ ID region code, e.g. -az1,
to allow reusable values across regions. Consider contention for resources and spot pricing in each AZ when selecting.
Useful in some regions when using only some AZs and you want to use the same ones across multiple accounts.		list(string)[]no
availability_zonesAWS Availability Zones in which to deploy multi-AZ resources.
Ignored if availability_zone_ids is set.
Can be the full name, e.g. us-east-1a, or just the part after the region, e.g. a to allow reusable values across regions.
If not provided, resources will be provisioned in every zone with a private subnet in the VPC.		list(string)[]no
aws_ssm_agent_enabledSet true to attach the required IAM policy for AWS SSM agent to each EC2 instance's IAM Roleboolfalseno
aws_sso_permission_sets_rbac(Not Recommended): AWS SSO (IAM Identity Center) permission sets in the EKS deployment account to add to aws-auth ConfigMap.
Unfortunately, aws-auth ConfigMap does not support SSO permission sets, so we map the generated
IAM Role ARN corresponding to the permission set at the time Terraform runs. This is subject to change
when any changes are made to the AWS SSO configuration, invalidating the mapping, and requiring a
terraform apply in this project to update the aws-auth ConfigMap and restore access.
list(object({
    aws_sso_permission_set = string
    groups                 = list(string)
  }))
[]no
aws_team_roles_rbacList of aws-team-roles (in the target AWS account) to map to Kubernetes RBAC groups.
list(object({
    aws_team_role = string
    groups        = list(string)
  }))
[]no
cluster_encryption_config_enabledSet to true to enable Cluster Encryption Configurationbooltrueno
cluster_encryption_config_kms_key_deletion_window_in_daysCluster Encryption Config KMS Key Resource argument - key deletion windows in days post destructionnumber10no
cluster_encryption_config_kms_key_enable_key_rotationCluster Encryption Config KMS Key Resource argument - enable kms key rotationbooltrueno
cluster_encryption_config_kms_key_idKMS Key ID to use for cluster encryption configstring""no
cluster_encryption_config_kms_key_policyCluster Encryption Config KMS Key Resource argument - key policystringnullno
cluster_encryption_config_resourcesCluster Encryption Config Resources to encrypt, e.g. ["secrets"]list(string)
[
  "secrets"
]
no
cluster_endpoint_private_accessIndicates whether or not the Amazon EKS private API server endpoint is enabled. Default to AWS EKS resource and it is falseboolfalseno
cluster_endpoint_public_accessIndicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is truebooltrueno
cluster_kubernetes_versionDesired Kubernetes master version. If you do not specify a value, the latest available version is usedstringnullno
cluster_log_retention_periodNumber of days to retain cluster logs. Requires enabled_cluster_log_types to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html.number0no
cluster_private_subnets_onlyWhether or not to enable private subnets or both public and private subnetsboolfalseno
colorThe cluster stage represented by a color; e.g. blue, greenstring""no
contextSingle object for setting entire context at once.
See description of individual variables for details.
Leave string and numeric variables as null to use default value.
Individual variable settings (non-null) override settings in context object,
except for attributes, tags, and additional_tag_map, which are merged.		any
{
  "additional_tag_map": {},
  "attributes": [],
  "delimiter": null,
  "descriptor_formats": {},
  "enabled": true,
  "environment": null,
  "id_length_limit": null,
  "label_key_case": null,
  "label_order": [],
  "label_value_case": null,
  "labels_as_tags": [
    "unset"
  ],
  "name": null,
  "namespace": null,
  "regex_replace_chars": null,
  "stage": null,
  "tags": {},
  "tenant": null
}
no
delimiterDelimiter to be used between ID elements.
Defaults to - (hyphen). Set to "" to use no delimiter at all.		stringnullno
deploy_addons_to_fargateSet to true (not recommended) to deploy addons to Fargate instead of initial node poolboolfalseno
descriptor_formatsDescribe additional descriptors to be output in the descriptors output map.
Map of maps. Keys are names of descriptors. Values are maps of the form
{<br/> format = string<br/> labels = list(string)<br/>}
(Type is any so the map values can later be enhanced to provide additional options.)
format is a Terraform format string to be passed to the format() function.
labels is a list of labels, in order, to pass to format() function.
Label values will be normalized before being passed to format() so they will be
identical to how they appear in id.
Default is {} (descriptors output will be empty).		any{}no
enabledSet to false to prevent the module from creating any resourcesboolnullno
enabled_cluster_log_typesA list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [api, audit, authenticator, controllerManager, scheduler]list(string)[]no
environmentID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT'stringnullno
fargate_profile_iam_role_kubernetes_namespace_delimiterDelimiter for the Kubernetes namespace in the IAM Role name for Fargate Profilesstring"-"no
fargate_profile_iam_role_permissions_boundaryIf provided, all Fargate Profiles IAM roles will be created with this permissions boundary attachedstringnullno
fargate_profilesFargate Profiles config
map(object({
    kubernetes_namespace = string
    kubernetes_labels    = map(string)
  }))
{}no
id_length_limitLimit id to this many characters (minimum 6).
Set to 0 for unlimited length.
Set to null for keep the existing setting, which defaults to 0.
Does not affect id_full.		numbernullno
karpenter_iam_role_enabledFlag to enable/disable creation of IAM role for EC2 Instance Profile that is attached to the nodes launched by Karpenterboolfalseno
label_key_caseControls the letter case of the tags keys (label names) for tags generated by this module.
Does not affect keys of tags passed in via the tags input.
Possible values: lower, title, upper.
Default value: title.		stringnullno
label_orderThe order in which the labels (ID elements) appear in the id.
Defaults to ["namespace", "environment", "stage", "name", "attributes"].
You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.		list(string)nullno
label_value_caseControls the letter case of ID elements (labels) as included in id,
set as tag values, and output by this module individually.
Does not affect values of tags passed in via the tags input.
Possible values: lower, title, upper and none (no transformation).
Set this to title and set delimiter to "" to yield Pascal Case IDs.
Default value: lower.		stringnullno
labels_as_tagsSet of labels (ID elements) to include as tags in the tags output.
Default is to include all labels.
Tags with empty values will not be included in the tags output.
Set to [] to suppress all generated tags.
Notes:
The value of the name tag, if included, will be the id, not the name.
Unlike other null-label inputs, the initial setting of labels_as_tags cannot be
changed in later chained modules. Attempts to change it will be silently ignored.		set(string)
[
  "default"
]
no
legacy_do_not_create_karpenter_instance_profileObsolete: The issues this was meant to mitigate were fixed in AWS Terraform Provider v5.43.0
and Karpenter v0.33.0. This variable will be removed in a future release.
Remove this input from your configuration and leave it at default.
Old description: When true (the default), suppresses creation of the IAM Instance Profile
for nodes launched by Karpenter, to preserve the legacy behavior of
the eks/karpenter component creating it.
Set to false to enable creation of the IAM Instance Profile, which
ensures that both the role and the instance profile have the same lifecycle,
and avoids AWS Provider issue #32671.
Use in conjunction with eks/karpenter component legacy_create_karpenter_instance_profile.		booltrueno
legacy_fargate_1_role_per_profile_enabledSet to false for new clusters to create a single Fargate Pod Execution role for the cluster.
Set to true for existing clusters to preserve the old behavior of creating
a Fargate Pod Execution role for each Fargate Profile.		booltrueno
managed_node_groups_enabledSet false to prevent the creation of EKS managed node groups.booltrueno
map_additional_aws_accounts(Obsolete) Additional AWS accounts to grant access to the EKS cluster.
This input is included to avoid breaking existing configurations that
supplied an empty list, but the list is no longer allowed to have entries.
(It is not clear that it worked properly in earlier versions in any case.)
This component now only supports EKS access entries, which require full principal ARNs.
This input is deprecated and will be removed in a future release.		list(string)[]no
map_additional_iam_rolesAdditional IAM roles to grant access to the cluster.
WARNING: Full Role ARN, including path, is required for rolearn.
In earlier versions (with aws-auth ConfigMap), only the path
had to be removed from the Role ARN. The path is now required.
username is now ignored. This input is planned to be replaced
in a future release with a more flexible input structure that consolidates
map_additional_iam_roles and map_additional_iam_users.
list(object({
    rolearn  = string
    username = optional(string)
    groups   = list(string)
  }))
[]no
map_additional_iam_usersAdditional IAM roles to grant access to the cluster.
username is now ignored. This input is planned to be replaced
in a future release with a more flexible input structure that consolidates
map_additional_iam_roles and map_additional_iam_users.
list(object({
    userarn  = string
    username = optional(string)
    groups   = list(string)
  }))
[]no
map_additional_worker_roles(Deprecated) AWS IAM Role ARNs of unmanaged Linux worker nodes to grant access to the EKS cluster.
In earlier versions, this could be used to grant access to worker nodes of any type
that were not managed by the EKS cluster. Now EKS requires that unmanaged worker nodes
be classified as Linux or Windows servers, in this input is temporarily retained
with the assumption that all worker nodes are Linux servers. (It is likely that
earlier versions did not work properly with Windows worker nodes anyway.)
This input is deprecated and will be removed in a future release.
In the future, this component will either have a way to separate Linux and Windows worker nodes,
or drop support for unmanaged worker nodes entirely.		list(string)[]no
nameID element. Usually the component or solution name, e.g. 'app' or 'jenkins'.
This is the only ID element not also included as a tag.
The "name" tag is set to the full id string. There is no tag with the value of the name input.		stringnullno
namespaceID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally uniquestringnullno
node_group_defaultsDefaults for node groups in the cluster
object({
    ami_release_version        = optional(string, null)
    ami_type                   = optional(string, null)
    attributes                 = optional(list(string), null)
    availability_zones         = optional(list(string)) # set to null to use var.availability_zones
    cluster_autoscaler_enabled = optional(bool, null)
    create_before_destroy      = optional(bool, null)
    desired_group_size         = optional(number, null)
    instance_types             = optional(list(string), null)
    kubernetes_labels          = optional(map(string), {})
    kubernetes_taints = optional(list(object({
      key    = string
      value  = string
      effect = string
    })), [])
    node_userdata = optional(object({
      before_cluster_joining_userdata = optional(string)
      bootstrap_extra_args            = optional(string)
      kubelet_extra_args              = optional(string)
      after_cluster_joining_userdata  = optional(string)
    }), {})
    kubernetes_version = optional(string, null) # set to null to use cluster_kubernetes_version
    max_group_size     = optional(number, null)
    min_group_size     = optional(number, null)
    resources_to_tag   = optional(list(string), null)
    tags               = optional(map(string), null)

    # block_device_map copied from cloudposse/terraform-aws-eks-node-group
    # Keep in sync via copy and paste, but make optional
    # Most of the time you want "/dev/xvda". For BottleRocket, use "/dev/xvdb".
    block_device_map = optional(map(object({
      no_device    = optional(bool, null)
      virtual_name = optional(string, null)
      ebs = optional(object({
        delete_on_termination = optional(bool, true)
        encrypted             = optional(bool, true)
        iops                  = optional(number, null)
        kms_key_id            = optional(string, null)
        snapshot_id           = optional(string, null)
        throughput            = optional(number, null) # for gp3, MiB/s, up to 1000
        volume_size           = optional(number, 50)   # disk  size in GB
        volume_type           = optional(string, "gp3")

        # Catch common camel case typos. These have no effect, they just generate better errors.
        # It would be nice to actually use these, but volumeSize in particular is a number here
        # and in most places it is a string with a unit suffix (e.g. 20Gi)
        # Without these defined, they would be silently ignored and the default values would be used instead,
        # which is difficult to debug.
        deleteOnTermination = optional(any, null)
        kmsKeyId            = optional(any, null)
        snapshotId          = optional(any, null)
        volumeSize          = optional(any, null)
        volumeType          = optional(any, null)
      }))
    })), null)

    # DEPRECATED: disk_encryption_enabled is DEPRECATED, use block_device_map instead.
    disk_encryption_enabled = optional(bool, null)
    # DEPRECATED: disk_size is DEPRECATED, use block_device_map instead.
    disk_size = optional(number, null)
  })
{
  "block_device_map": {
    "/dev/xvda": {
      "ebs": {
        "encrypted": true,
        "volume_size": 20,
        "volume_type": "gp2"
      }
    }
  },
  "desired_group_size": 1,
  "instance_types": [
    "t3.medium"
  ],
  "kubernetes_version": null,
  "max_group_size": 100
}
no
node_groupsList of objects defining a node group for the cluster
map(object({
    # EKS AMI version to use, e.g. "1.16.13-20200821" (no "v").
    ami_release_version = optional(string, null)
    # Type of Amazon Machine Image (AMI) associated with the EKS Node Group
    ami_type = optional(string, null)
    # Additional attributes (e.g. 1) for the node group
    attributes = optional(list(string), null)
    # will create 1 auto scaling group in each specified availability zone
    # or all AZs with subnets if none are specified anywhere
    availability_zones = optional(list(string), null)
    # Whether to enable Node Group to scale its AutoScaling Group
    cluster_autoscaler_enabled = optional(bool, null)
    # True to create new node_groups before deleting old ones, avoiding a temporary outage
    create_before_destroy = optional(bool, null)
    # Desired number of worker nodes when initially provisioned
    desired_group_size = optional(number, null)
    # Set of instance types associated with the EKS Node Group. Terraform will only perform drift detection if a configuration value is provided.
    instance_types = optional(list(string), null)
    # Key-value mapping of Kubernetes labels. Only labels that are applied with the EKS API are managed by this argument. Other Kubernetes labels applied to the EKS Node Group will not be managed
    kubernetes_labels = optional(map(string), null)
    # List of objects describing Kubernetes taints.
    kubernetes_taints = optional(list(object({
      key    = string
      value  = string
      effect = string
    })), null)
    node_userdata = optional(object({
      before_cluster_joining_userdata = optional(string)
      bootstrap_extra_args            = optional(string)
      kubelet_extra_args              = optional(string)
      after_cluster_joining_userdata  = optional(string)
    }), {})
    # Desired Kubernetes master version. If you do not specify a value, the latest available version is used
    kubernetes_version = optional(string, null)
    # The maximum size of the AutoScaling Group
    max_group_size = optional(number, null)
    # The minimum size of the AutoScaling Group
    min_group_size = optional(number, null)
    # List of auto-launched resource types to tag
    resources_to_tag = optional(list(string), null)
    tags             = optional(map(string), null)

    # block_device_map copied from cloudposse/terraform-aws-eks-node-group
    # Keep in sync via copy and paste, but make optional.
    # Most of the time you want "/dev/xvda". For BottleRocket, use "/dev/xvdb".
    block_device_map = optional(map(object({
      no_device    = optional(bool, null)
      virtual_name = optional(string, null)
      ebs = optional(object({
        delete_on_termination = optional(bool, true)
        encrypted             = optional(bool, true)
        iops                  = optional(number, null)
        kms_key_id            = optional(string, null)
        snapshot_id           = optional(string, null)
        throughput            = optional(number, null) # for gp3, MiB/s, up to 1000
        volume_size           = optional(number, 20)   # Disk size in GB
        volume_type           = optional(string, "gp3")

        # Catch common camel case typos. These have no effect, they just generate better errors.
        # It would be nice to actually use these, but volumeSize in particular is a number here
        # and in most places it is a string with a unit suffix (e.g. 20Gi)
        # Without these defined, they would be silently ignored and the default values would be used instead,
        # which is difficult to debug.
        deleteOnTermination = optional(any, null)
        kmsKeyId            = optional(any, null)
        snapshotId          = optional(any, null)
        volumeSize          = optional(any, null)
        volumeType          = optional(any, null)
      }))
    })), null)

    # DEPRECATED:
    # Enable disk encryption for the created launch template (if we aren't provided with an existing launch template)
    # DEPRECATED: disk_encryption_enabled is DEPRECATED, use block_device_map instead.
    disk_encryption_enabled = optional(bool, null)
    # Disk size in GiB for worker nodes. Terraform will only perform drift detection if a configuration value is provided.
    # DEPRECATED: disk_size is DEPRECATED, use block_device_map instead.
    disk_size = optional(number, null)

  }))
{}no
oidc_provider_enabledCreate an IAM OIDC identity provider for the cluster, then you can create IAM roles to associate with a service account in the cluster, instead of using kiam or kube2iam. For more information, see https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.htmlbooltrueno
public_access_cidrsIndicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled. EKS defaults this to a list with 0.0.0.0/0.list(string)
[
  "0.0.0.0/0"
]
no
regex_replace_charsTerraform regular expression (regex) string.
Characters matching the regex will be removed from the ID elements.
If not set, "/[^a-zA-Z0-9-]/" is used to remove all characters other than hyphens, letters and digits.		stringnullno
regionAWS Regionstringn/ayes
stageID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release'stringnullno
subnet_type_tag_keyThe tag used to find the private subnets to find by availability zone. If null, will be looked up in vpc outputs.stringnullno
tagsAdditional tags (e.g. {'BusinessUnit': 'XYZ'}).
Neither the tag keys nor the tag values will be modified by this module.		map(string){}no
tenantID element _(Rarely used, not included by default)_. A customer identifier, indicating who this instance of a resource is forstringnullno
vpc_component_nameThe name of the vpc componentstring"vpc"no

Outputs​

NameDescription
availability_zonesAvailability Zones in which the cluster is provisioned
eks_addons_versionsMap of enabled EKS Addons names and versions
eks_auth_worker_rolesList of worker IAM roles that were included in the auth-map ConfigMap.
eks_cluster_arnThe Amazon Resource Name (ARN) of the cluster
eks_cluster_certificate_authority_dataThe Kubernetes cluster certificate authority data
eks_cluster_endpointThe endpoint for the Kubernetes API server
eks_cluster_idThe name of the cluster
eks_cluster_identity_oidc_issuerThe OIDC Identity issuer for the cluster
eks_cluster_managed_security_group_idSecurity Group ID that was created by EKS for the cluster. EKS creates a Security Group and applies it to ENI that is attached to EKS Control Plane master nodes and to any managed workloads
eks_cluster_versionThe Kubernetes server version of the cluster
eks_managed_node_workers_role_arnsList of ARNs for workers in managed node groups
eks_node_group_arnsList of all the node group ARNs in the cluster
eks_node_group_countCount of the worker nodes
eks_node_group_idsEKS Cluster name and EKS Node Group name separated by a colon
eks_node_group_role_namesList of worker nodes IAM role names
eks_node_group_statusesStatus of the EKS Node Group
fargate_profile_role_arnsFargate Profile Role ARNs
fargate_profile_role_namesFargate Profile Role names
fargate_profilesFargate Profiles
karpenter_iam_role_arnKarpenter IAM Role ARN
karpenter_iam_role_nameKarpenter IAM Role name
vpc_cidrThe CIDR of the VPC where this cluster is deployed.

References​

CHANGELOG​

Release 1.468.0​

PR #1072

Bugfix:

  • Correctly map AWS SSO Permission Sets referenced by aws_sso_permission_sets_rbac to IAM Role ARNs.
  • Broken in Release 1.431.1: Update to use AWS Auth API

Release 1.467.0​

PR #1071

Bugfix: Update cloudposse/eks-node-group/aws to v3.0.1.

  • Fixes failure to create userdata for AL2 and Windows when using it to run bootstrap.sh.

Release 1.465.0​

Components PR #1069

Update cloudposse/eks-node-group/aws to v3.0.0

Release 1.455.1​

Components PR #1057

Fixed "Invalid count argument" argument when creating new cluster

Release 1.452.0​

Components PR #1046

Added support for passing extra arguments to kubelet and other startup modifications supported by EKS on Amazon Linux 2 via the bootsrap.sh script.

This support should be considered an alpha version, as it may change when support for Amazon Linux 2023 is added, and does not work with Bottlerocket.

Release 1.431.1: Breaking Changes​

Components PR #1033

Major Breaking Changes​

Major Breaking Changes, Manual Intervention Required

This release includes a major breaking change that requires manual intervention to migrate existing clusters. The change is necessary to support the new AWS Access Control API, which is more secure and more reliable than the old aws-auth ConfigMap.

This release drops support for the aws-auth ConfigMap and switches to managing access control with the new AWS Access Control API. This change allows for more secure and reliable access control, and removes the requirement that Terraform operations on the EKS cluster itself require network access to the EKS control plane.

In this release, this component only supports assigning "team roles" to Kubernetes RBAC groups. Support for AWS EKS Access Policies is not yet implemented. However, if you specify system:masters as a group, that will be translated into assigning the AmazonEKSClusterAdminPolicy to the role. Any other system:* group will cause an error.

Network Access Considerations

Previously, this component required network access to the EKS control plane to manage the aws-auth ConfigMap. This meant having the EKS control plane accessible from the public internet, or using a bastion host or VPN to access the control plane. With the new AWS Access Control API, Terraform operations on the EKS cluster no longer require network access to the EKS control plane.

This may seem like it makes it easier to secure the EKS control plane, but Terraform users will still require network access to the EKS control plane to manage any deployments or other Kubernetes resources in the cluster. This means that this upgrade does not substantially change the need for network access.

Minor Changes​

With the fixes included and AWS Terraform Provider v5.43.0 and Karpenter v0.33.0, the legacy_do_not_create_karpenter_instance_profile is now obsolete. After upgrading both this component and the eks/karpenter component, if you had it in your configuration, you can remove it. If you had previously set it to false, removing it may cause an error when you apply the changes. If you see an error about the aws_iam_instance_profile resource being destroyed (cannot be destroyed because it is in use, has dependencies, and/or has role attached), you can simply remove the resource from the Terraform state with [atmos] terraform state rm, because it will be managed by the Karpenter controller instead of Terraform.

Access Control API Migration Procedure​

Full details of the migration process can be found in the cloudposse/terraform-aws-eks-cluster migration document. This section is a streamlined version for users of this eks/cluster component.

info

The commands below assume the component is named "eks/cluster". If you are using a different name, replace "eks/cluster" with the correct component name.

Prepare for Migration​

Make sure you have kubectl access to the cluster, preferably using the aws eks get-token command configured into your $KUBECONFIG file. Geodesic users can usually set this up with

atmos aws eks update-kubeconfig eks/cluster -s=<stack-name>
## or
set-cluster <tenant>-<region>-<stage>

Where <tenant> is the "tenant" name, a.k.a. the "org" name, e.g. "core", and should be omitted (along with the hyphen) if your organization does not use a tenant name. <region> is the AWS region abbreviation your organization is using, e.g. "usw2" or "uw2", and <stage> is the "stage" or "account" name, e.g. "auto" or "prod".

Test your access with kubectl

## check if you have any access at all. Should output "yes".
kubectl auth can-i -A create selfsubjectaccessreviews.authorization.k8s.io

## Do you have full cluster administrator access?
kubectl auth can-i '*' '*'

## Show me what I can and cannot do (if `rakkess` is installed)
rakkess

Migrate​
  1. Update the component (already done if you see this document).
  2. Run atmos terraform plan eks/cluster -s <stack_name>

See this error:

To work with module.eks_cluster.kubernetes_config_map.aws_auth[0] (orphan) its original provider configuration

Note, in other documentation, the exact "address" of the orphaned resource may be different, and the documentation may say to refer to the address of the resource in the error message. In this case, because we are using this component as the root module, the address should be exactly as shown above. (Possibly ending with aws_auth_ignore_changes[0] instead of aws_auth[0].)

  1. Remove the orphaned resource from the state file with
atmos terraform state rm eks/cluster 'module.eks_cluster.kubernetes_config_map.aws_auth[0]' -s <stack_name>
  1. atmos terraform plan eks/cluster -s <stack_name>

Verify:

  • module.eks_cluster.aws_eks_cluster.default[0] will be updated in-place
    • access_config.authentication_mode = "CONFIG_MAP" -> "API_AND_CONFIG_MAP"

Stop and ask for help if you see module.eks_cluster.aws_eks_cluster.default[0] will be destroyed. Expect to see a lot of IAM changes due to the potential for the EKS OIDC thumbprint to change, and a lot of aws_eks_access_entry additions. You may also see:

  • aws_security_group_rule resources replaced by aws_vpc_security_group_ingress_rule resources
  • null_resource resources destroyed
  1. Apply the plan with atmos terraform apply eks/cluster -s <stack_name> --from-plan

EXPECT AN ERROR. Something like:

│ Error: creating EKS Access Entry
(eg-core-usw2-auto-eks-cluster:arn:aws:iam::123456789012:role/eg-core-gbl-auto-terraform): operation error EKS: CreateAccessEntry, https response error StatusCode: 409, RequestID: 97a40994-4223-4af1-977e-42ec57eb3ad6, ResourceInUseException: The specified access entry resource is already in use on this cluster.
│
│   with module.eks_cluster.aws_eks_access_entry.map["arn:aws:iam::123456789012:role/eg-core-gbl-auto-terraform"],
│   on .terraform/modules/eks_cluster/auth.tf line 60, in resource "aws_eks_access_entry" "map":
│   60: resource "aws_eks_access_entry" "map" {

This is expected. The access entry is something we want to control, but a duplicate is automatically created by AWS during the conversion. Import the created entry. You may get other errors, but they are likely transient and will be fixed automatically after fixing this one.

The access entry ID to import is given in the error message in parentheses. In the example above, the ID is eg-core-usw2-auto-eks-cluster:arn:aws:iam::123456789012:role/eg-core-gbl-auto-terraform.

The Terraform resource address for the resource will also be in the error message: it is the part after "with". In the example above, the address is

module.eks_cluster.aws_eks_access_entry.map["arn:aws:iam::123456789012:role/eg-core-gbl-auto-terraform"]

Import the resource with

atmos terraform import eks/cluster '<resource address>' '<access entry ID>' -s <stack_name>

It is critical to use single quotes around the resource address and access entry ID to prevent the shell from interpreting the square brackets and colons and to preserve the double quotes in the resource address.

After successfully importing the resource, run

atmos terraform apply eks/cluster -s <stack_name>`

to apply tags to the entry and finish up any changes interrupted by the error. It should apply cleanly this time.

Verify​

Verify that you still have access to the cluster with kubectl, just as you did in the "Prepare" section.

Cleanup​

Either one cluster at a time, or later in an organization-wide cleanup, migrate all clusters from API_AND_CONFIG_MAP to API authentication mode.

At this point you have both the old and new access control methods enabled, but nothing is managing the aws-auth ConfigMap. The aws-auth ConfigMap has been abandoned by this module and will no longer have entries added or, crucially, removed. In order to remove this lingering unmanaged grant of access, migrate the cluster to API authentication mode, and manually remove the aws-auth ConfigMap.

  • Update the access.config.authentication_mode to "API" in your configuration:

    access_config:
    authentication_mode: API

    and run atmos terraform apply again. This will cause EKS to ignore the aws-auth ConfigMap, but will not remove it. Again, this will cause a lot of IAM changes due to the potential for the EKS OIDC thumbprint to change, but this is not a problem.

  • Manually remove the aws-auth ConfigMap. You can do this with kubectl delete configmap aws-auth --namespace kube-system. This will not affect the cluster, because it is now being managed by the new access control API, but it will reduce the possibility of confusion in the future.

End of Access Control API Migration​

Changes in v1.349.0​

Components PR #910

Bug fix and updates to Changelog, no action required.

Fixed: Error about managed node group ARNs list being null, which could happen when adding a managed node group to an existing cluster that never had one.

Changes in v1.303.0​

Components PR #852

This is a bug fix and feature enhancement update. No action is necessary to upgrade. However, with the new features and new recommendations, you may want to change your configuration.

Previously, we recommended deploying Karpenter to Fargate and not provisioning any nodes. However, this causes issues with add-ons that require compute power to fully initialize, such as coredns, and it can reduce the cluster to a single node, removing the high availability that comes from having a node per Availability Zone and replicas of pods spread across those nodes.

As a result, we now recommend deploying a minimal node group with a single instance (currently recommended to be a c7a.medium) in each of 3 Availability Zones. This will provide the compute power needed to initialize add-ons, and will provide high availability for the cluster. As a bonus, it will also remove the need to deploy Karpenter to Fargate.

NOTE about instance type: The c7a.medium instance type is relatively new. If you have deployed an old version of our ServiceControlPolicy DenyEC2NonNitroInstances, DenyNonNitroInstances (obsolete, replaced by DenyEC2NonNitroInstances), and/or DenyEC2InstancesWithoutEncryptionInTransit, you will want to update them to v0.14.1 or choose a different instance type.

Migration procedure​

To perform the recommended migration, follow these steps:

1. Deploy a minimal node group, move addons to it​

Change your eks/cluster configuration to set deploy_addons_to_fargate: false.

Add the following to your eks/cluster configuration, but copy the block device name, volume size, and volume type from your existing Karpenter provisioner configuration. Also select the correct ami_type according to the ami_family in your Karpenter provisioner configuration.

node_groups:
  # will create 1 node group for each item in map
  # Provision a minimal static node group for add-ons and redundant replicas
  main:
    # EKS AMI version to use, e.g. "1.16.13-20200821" (no "v").
    ami_release_version: null
    # Type of Amazon Machine Image (AMI) associated with the EKS Node Group
    # Typically AL2_x86_64 or BOTTLEROCKET_x86_64
    ami_type: BOTTLEROCKET_x86_64
    # Additional name attributes (e.g. `1`) for the node group
    attributes: []
    # will create 1 auto scaling group in each specified availability zone
    # or all AZs with subnets if none are specified anywhere
    availability_zones: null
    # Whether to enable Node Group to scale its AutoScaling Group
    cluster_autoscaler_enabled: false
    # True (recommended) to create new node_groups before deleting old ones, avoiding a temporary outage
    create_before_destroy: true
    # Configure storage for the root block device for instances in the Auto Scaling Group
    # For Bottlerocket, use /dev/xvdb. For all others, use /dev/xvda.
    block_device_map:
      "/dev/xvdb":
        ebs:
          volume_size: 125 # in GiB
          volume_type: gp3
          encrypted: true
          delete_on_termination: true
    # Set of instance types associated with the EKS Node Group. Terraform will only perform drift detection if a configuration value is provided.
    instance_types:
      - c6a.large
    # Desired number of worker nodes when initially provisioned
    desired_group_size: 3
    max_group_size: 3
    min_group_size: 3
    resources_to_tag:
      - instance
      - volume
    tags: null

You do not need to apply the above changes yet, although you can if you want to. To reduce overhead, you can apply the changes in the next step.

2. Move Karpenter to the node group, remove legacy support​

Delete the fargate_profiles section from your eks/cluster configuration, or at least remove the karpenter profile from it. Disable legacy support by adding:

legacy_fargate_1_role_per_profile_enabled: false
2.a Optional: Move Karpenter instance profile to eks/cluster component​

If you have the patience to manually import and remove a Terraform resource, you should move the Karpenter instance profile to the eks/cluster component. This fixes an issue where the Karpenter instance profile could be broken by certain sequences of Terraform operations. However, if you have multiple clusters to migrate, this can be tedious, and the issue is not a serious one, so you may want to skip this step.

To do this, add the following to your eks/cluster configuration:

legacy_do_not_create_karpenter_instance_profile: false

BEFORE APPLYING CHANGES: Run atmos terraform plan (with the appropriate arguments) to see the changes that will be made. Among the resources to be created will be aws_iam_instance_profile.default[0]. Using the same arguments as before, run atmos, but replace plan with import 'aws_iam_instance_profile.default[0]' <profile-name>, where <profile-name> is the name of the profile the plan indicated it would create. It will be something like <cluster-name>-karpenter.

NOTE: If you perform this step, you must also perform 3.a below.

2.b Apply the changes​

Apply the changes with atmos terraform apply.

3. Upgrade Karpenter​

Upgrade the eks/karpenter component to the latest version. Follow the upgrade instructions to enable the new karpenter-crd chart by setting crd_chart_enabled: true.

Upgrade to at least Karpenter v0.30.0, which is the first version to support factoring in the existing node group when determining the number of nodes to provision. This will prevent Karpenter from provisioning nodes when they are not needed because the existing node group already has enough capacity. Be careful about upgrading to v0.32.0 or later, as that version introduces significant breaking changes. We recommend updating to v0.31.2 or later versions of v0.31.x, but not v0.32.0 or later, as a first step. This provides a safe (revertible) upgrade path to v0.32.0 or later.

3.a Finish Move of Karpenter instance profile to eks/cluster component​

If you performed step 2.a above, you must also perform this step. If you did not perform step 2.a, you must NOT perform this step.

In the eks/karpenter stack, set legacy_create_karpenter_instance_profile: false.

BEFORE APPLYING CHANGES: Remove the Karpenter instance profile from the Terraform state, since it is now managed by the eks/cluster component, or else Terraform will delete it.

atmos terraform state eks/karpenter rm 'aws_iam_instance_profile.default[0]' -s=<stack-name>
3.b Apply the changes​

Apply the changes with atmos terraform apply.

Changes included in v1.303.0​

This is a bug fix and feature enhancement update. No action is necessary to upgrade.

Bug Fixes​

  • Timeouts for Add-Ons are now honored (they were being ignored)
  • If you supply a service account role ARN for an Add-On, it will be used, and no new role will be created. Previously it was used, but the component created a new role anyway.
  • The EKS EFS controller add-on cannot be deployed to Fargate, and enabling it along with deploy_addons_to_fargate will no longer attempt to deploy EFS to Fargate. Note that this means to use the EFS Add-On, you must create a managed node group. Track the status of this feature with this issue.
  • If you are using an old VPC component that does not supply az_private_subnets_map, this module will now use the older the private_subnet_ids output.

Add-Ons have enabled option​

The EKS Add-Ons now have an optional "enabled" flag (defaults to true) so that you can selectively disable them in a stack where the inherited configuration has them enabled.

Upgrading to v1.270.0​

Components PR #795

Removed identity roles from cluster RBAC (aws-auth ConfigMap)​

Previously, this module added identity roles configured by the aws_teams_rbac input to the aws-auth ConfigMap. This never worked, and so now aws_teams_rbac is ignored. When upgrading, you may see these roles being removed from the aws-auth: this is expected and harmless.

Better support for Manged Node Group Block Device Specifications​

Previously, this module only supported specifying the disk size and encryption state for the root volume of Managed Node Groups. Now, the full set of block device specifications is supported, including the ability to specify the device name. This is particularly important when using BottleRocket, which uses a very small root volume for storing the OS and configuration, and exposes a second volume (/dev/xvdb) for storing data.

Block Device Migration​

Almost all of the attributes of node_groups and node_group_defaults are now optional. This means you can remove from your configuration any attributes that previously you were setting to null.

The disk_size and disk_encryption_enabled attributes are deprecated. They only apply to /dev/xvda, and only provision a gp2 volume. In order to provide backwards compatibility, they are still supported, and, when specified, cause the new block_device_map attribute to be ignored.

The new block_device_map attribute is a map of objects. The keys are the names of block devices, and the values are objects with the attributes from the Terraform launch_template.block-devices resource.

Note that the new default, when none of block_device_map, disk_size, or disk_encryption_enabled are specified, is to provision a 20GB gp3 volume for /dev/xvda, with encryption enabled. This is a change from the previous default, which provisioned a gp2 volume instead.

Support for EFS add-on​

This module now supports the EFS CSI driver add-on, in very much the same way as it supports the EBS CSI driver add-on. The only difference is that the EFS CSI driver add-on requires that you first provision an EFS file system.

Migration from eks/efs-controller to EFS CSI Driver Add-On​

If you are currently using the eks/efs-controller module, you can migrate to the EFS CSI Driver Add-On by following these steps:

  1. Remove or scale to zero Pods any Deployments using the EFS file system.
  2. Remove (terraform destroy) the eks/efs-controller module from your cluster. This will also remove the efs-sc StorageClass.
  3. Use the eks/storage-class module to create a replacement EFS StorageClass efs-sc. This component is new and you may need to add it to your cluster.
  4. Deploy the EFS CSI Driver Add-On by adding aws-efs-csi-driver to the addons map (see README).
  5. Restore the Deployments you modified in step 1.

More options for specifying Availability Zones​

Previously, this module required you to specify the Availability Zones for the cluster in one of two ways:

  1. Explicitly, by providing the full AZ names via the availability_zones input
  2. Implicitly, via private subnets in the VPC

Option 2 is still usually the best way, but now you have additional options:

  • You can specify the Availability Zones via the availability_zones input without specifying the full AZ names. You can just specify the suffixes of the AZ names, and the module will find the full names for you, using the current region. This is useful for using the same configuration in multiple regions.
  • You can specify Availability Zone IDs via the availability_zone_ids input. This is useful to ensure that clusters in different accounts are nevertheless deployed to the same Availability Zones. As with the availability_zones input, you can specify the suffixes of the AZ IDs, and the module will find the full IDs for you, using the current region.

Support for Karpenter Instance Profile​

Previously, this module created an IAM Role for instances launched by Karpenter, but did not create the corresponding Instance Profile, which was instead created by the eks/karpenter component. This can cause problems if you delete and recreate the cluster, so for new clusters, this module can now create the Instance Profile as well.

Because this is disruptive to existing clusters, this is not enabled by default. To enable it, set the legacy_do_not_create_karpenter_instance_profile input to false, and also set the eks/karpenter input legacy_create_karpenter_instance_profile to false.

Upgrading to v1.250.0​

Components PR #723

Improved support for EKS Add-Ons​

This has improved support for EKS Add-Ons.

Configuration and Timeouts​

The addons input now accepts a configuration_values input to allow you to configure the add-ons, and various timeout inputs to allow you to fine-tune the timeouts for the add-ons.

Automatic IAM Role Creation​

If you enable aws-ebs-csi-driver or vpc-cni add-ons, the module will automatically create the required Service Account IAM Role and attach it to the add-on.

Add-Ons can be deployed to Fargate​

If you are using Karpenter and not provisioning any nodes with this module, the coredns and aws-ebs-csi-driver add-ons can be deployed to Fargate. (They must be able to run somewhere in the cluster or else the deployment will fail.)

To cause the add-ons to be deployed to Fargate, set the deploy_addons_to_fargate input to true.

Note about CoreDNS: If you want to deploy CoreDNS to Fargate, as of this writing you must set the configuration_values input for CoreDNS to '{"computeType": "Fargate"}'. If you want to deploy CoreDNS to EC2 instances, you must NOT include the computeType configuration value.

Availability Zones implied by Private Subnets​

You can now avoid specifying Availability Zones for the cluster anywhere. If all of the possible Availability Zones inputs are empty, the module will use the Availability Zones implied by the private subnets. That is, it will deploy the cluster to all of the Availability Zones in which the VPC has private subnets.

Optional support for 1 Fargate Pod Execution Role per Cluster​

Previously, this module created a separate Fargate Pod Execution Role for each Fargate Profile it created. This is unnecessary, excessive, and can cause problems due to name collisions, but is otherwise merely inefficient, so it is not important to fix this on existiong, working clusters. This update brings a feature that causes the module to create at most 1 Fargate Pod Execution Role per cluster.

This change is recommended for all NEW clusters, but only NEW clusters. Because it is a breaking change, it is not enabled by default. To enable it, set the legacy_fargate_1_role_per_profile_enabled variable to false.

WARNING: If you enable this feature on an existing cluster, and that cluster is using Karpenter, the update could destroy all of your existing Karpenter-provisioned nodes. Depending on your Karpenter version, this could leave you with stranded EC2 instances (still running, but not managed by Karpenter or visible to the cluster) and an interruption of service, and possibly other problems. If you are using Karpenter and want to enable this feature, the safest way is to destroy the existing cluster and create a new one with this feature enabled.