Skip to main content
Sponsor @cloudposse

Decide on AWS Account VPC Subnet CIDR Strategy

Problem​

We need to devise a subnet allocation scheme tolerant of multiple accounts operating in multiple regions that do not conflict with any other ranges which may need to be peered in the future.

General Considerations​

  • Having unique, non-overlapping VPC CIDRs makes connecting clusters to each other much easier

  • Each VPC must be subdivided into several non-overlapping subnet ranges to provide public and private address spaces across multiple availability zones

  • ALBs need a minimum of 2 subnets allocated

EKS Considerations​

  • Using Amazon’s CNI, each Kubernetes pod gets its own IP in the subnet, and additional IPs are reserved so they are immediately available for new pods when they are launched

  • You will need a lot more IPs than you anticipate due to performance optimizations in how CNIs are managed by EKS https://betterprogramming.pub/amazon-eks-is-eating-my-ips-e18ea057e045 https://medium.com/codex/kubernetes-cluster-running-out-of-ip-addresses-on-aws-eks-c7b8e5dd8606

  • AWS supports the eksctl tool (we do not). Their default recommendation is:

    The default VPC CIDR used by eksctl is 192.168.0.0/16. It is divided into 8 (/19) subnets (3 private, 3 public & 2 reserved).

  • EKS clusters limit the number of pods based on the number of ENIs available per instance type.

  • Kubernetes has limits, but those are pretty high. The reality is most clusters operate at a much smaller scale. At v1.18, Kubernetes supports clusters with up to 5000 nodes. More specifically, we support configurations that meet all of the following criteria:

  • No more than 5000 nodes

  • No more than 150000 total pods

  • No more than 300000 total containers

  • No more than 100 pods per node

  • EKS will use either 10.100.0.0/16 or 172.20.0.0/16 for cluster Services, so avoiding those ranges will avoid some problems with inter-cluster routing

Use CIDR ranges smaller than a /19 at your own risk.

Cloud Posse does not take responsibility for any EKS cluster issues related to underprovisioning CIDR ranges.

Our standard recommendation​

  • Each account gets it's own /16 (65,534 usable IPs) (or /15 = 2 x /16 for more than 4 total regions), consecutively numbered, starting with 10.101.0.0

  • Each region in an account gets 1 x /18 (16,382 usable IPs), usually allocated as 1 or 2 countries/legislative areas per account, each with 2 regions for DR/failover

  • Each region allocates 6 x /21 (2,046 usable IPs) subnets (3 AZ * (public + private)) for EKS.

  • Any additional “single purpose” subnets in a region should be /24 (254 usable IPs)

Further reading:

CIDR Subnet Table​

Subnet MaskCIDR PrefixTotal IP AddressesUsable IP AddressesNumber of /24 networks
255.255.255.255/32111/256th
255.255.255.254/3122*1/128th
255.255.255.252/30421/64th
255.255.255.248/29861/32nd
255.255.255.240/2816141/16th
255.255.255.224/2732301/8th
255.255.255.192/2664621/4th
255.255.255.128/251281261 half
255.255.255.0/242562541
255.255.254.0/235125102
255.255.252.0/221,0241,0224
255.255.248.0/212,0482,0468
255.255.240.0/204,0964,09416
255.255.224.0/198,1928,19032
255.255.192.0/1816,38416,38264
255.255.128.0/1732,76832,766128
255.255.0.0/1665,53665,534256
255.254.0.0/15131,072131,070512
255.252.0.0/14262,144262,1421024
255.248.0.0/13524,288524,2862048
255.240.0.0/121,048,5761,048,5744096
255.224.0 0/112,097,1522,097,1508192
255.192.0.0/104,194,3044,194,30216,384
255.128.0.0/98,388,6088,388,60632,768
255.0.0.0/816,777,21616,777,21465,536
254.0.0.0/733,554,43233,554,430131,072
252.0.0.0/667,108,86467,108,862262,144
248.0.0.0/5134,217,728134,217,7261,048,576
240.0.0.0/4268,435,456268,435,4542,097,152
224.0.0.0/3536,870,912536,870,9104,194,304
192.0.0.0/21,073,741,8241,073,741,8228,388,608
128.0.0.0/12,147,483,6482,147,483,64616,777,216
0.0.0.0/04,294,967,2964,294,967,29433,554,432