AWS EKS – Part 25 – Kubernetes Networking and AWS VPC CNI

Saeid Bostandoust20 May 2024

AWS VPC CNI is the default CNI “Container Network Interface” in EKS, and it’s installed automatically when the cluster is deployed for the first time. Kubernetes Networking is separated into two main topics, Pod Networking and Service Networking, in which CNI plugins are responsible for providing and managing Pods and container networking needs. In this lesson, you will learn more about Kubernetes networking and how to use AWS VPC CNI and customise its configuration to achieve special needs.

Follow our social media:

https://linkedin.com/in/ssbostan

https://linkedin.com/company/kubedemy

https://youtube.com/@kubedemy

https://telegram.me/kubedemy

Register for the FREE EKS Tutorial:

If you want to access the course materials, register from the following link:

Register for the FREE AWS EKS Black Belt Course

Kubernetes Networking:

Service Networking: It belongs to the Kubernetes Service resource and is used to forward traffic from an IP address within the Service CIDR into an IP address within the Pod CIDR to reach the proper Pod/Container. This network has no actual interfaces and is managed on-fly by Kube-proxy through itself, IPTables, NFTables, or IPVS. Some networking plugins, like Cilium, can also replace it. In EKS, you can set the Service CIDR when creating a cluster, and it cannot be changed after cluster creation. As a part of Service Networking, CoreDNS “or kube-dns in the past” provides DNS service and allows Pods and entities to resolve Service CIDR IP addresses using DNS names. Kube-proxy and CoreDNS will be deployed into worker nodes as soon as you deploy them.

Pod Networking: It belongs to every Pod in your cluster and is managed by CNI plugins, such as VPC CNI, Cilium, Calico, etc. The CNI plugin is responsible for creating a proper network interface and assigning it to the Pod when it’s created. This network has actual interfaces, and the CNI plugin creates an interface “mostly veth” and assigns it the Pod. As a matter of fact, it will be assigned to the Pod’s pause container. With this network, containers in the same Pod, Pods in the same worker node and Pods in different worker nodes can communicate without using NAT or any other translation. In EKS, AWS VPC CNI is the default CNI and is deployed automatically when the cluster is created. Unlike Service Networking, Pod Networking can be changed even after cluster creation. The default VPC CNI can be replaced by other CNI plugins like Cilium and Calico using the BYOCNI method, or those CNIs can be installed on top of VPC CNI using the CNI chaining. Both of these solutions will be covered in future articles.

Kubernetes Networking Components:

Kube-proxy: deployed as a DaemonSet in every worker node.
CoreDNS: deployed as a Deployment with one or more replicas.
VPC CNI: deployed as a DaemonSet in every worker node.

EKS Networking deployment procedure:

Install Networking components as managed EKS addons.
Achieve more security using IRSA deployment for AWS VPC CNI.
Customise AWS VPC CNI configurations to unleash features.

Step 1 – Install EKS Networking Components:

As mentioned above, AWS VPC CNI and other Kubernetes networking components are installed by default when you create an EKS cluster, but they are not part of EKS-managed addons. We must install them as managed addons so we can upgrade and customise them through EKS APIs. To do so, run the following commands.

To learn about deploying EKS addons, read the following article:

AWS EKS – Part 22 – Deploy and Manage Amazon EKS Cluster Addons

To install AWS VPC CNI:

# Create addon configuration.
cat <<EOF > vpc-cni-configuration.yaml
tolerations:
  - operator: Exists
EOF

aws eks create-addon \
  --cluster-name kubedemy \
  --addon-name vpc-cni \
  --addon-version v1.18.1-eksbuild.1 \
  --resolve-conflicts OVERWRITE \
  --configuration-values file://vpc-cni-configuration.yaml \
  --tags owner=kubedemy

To install Kube-proxy:

# Create addon configuration.
cat <<EOF > kube-proxy-configuration.yaml
mode: ipvs
EOF

aws eks create-addon \
  --cluster-name kubedemy \
  --addon-name kube-proxy \
  --addon-version v1.29.3-eksbuild.2 \
  --resolve-conflicts OVERWRITE \
  --configuration-values file://kube-proxy-configuration.yaml \
  --tags owner=kubedemy

To install CoreDNS:

# Create addon configuration.
cat <<EOF > coredns-configuration.yaml
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: node.kubernetes.io/scope
            operator: In
            values:
            - system
tolerations:
  - key: CriticalAddonsOnly
    operator: Exists
EOF

aws eks create-addon \
  --cluster-name kubedemy \
  --addon-name coredns \
  --addon-version v1.11.1-eksbuild.8 \
  --resolve-conflicts OVERWRITE \
  --configuration-values file://coredns-configuration.yaml \
  --tags owner=kubedemy

Step 2 – Install AWS VPC CNI with IRSA:

AWS VPC CNI needs AmazonEKS_CNI_Policy IAM policy to be able to create and attach network interfaces and assign IP addresses to EC2 instances. We usually attach this policy to the worker node IAM role, meaning every Pod in the cluster can access those resources and manipulate network interfaces and IP addresses. To prevent such behaviours, we must deploy addons using IRSA so that no one except the addon Pods can access AWS resources. This is a best practice for production environments.

AWS EKS – Part 13 – Setup IAM Roles for Service Accounts (IRSA)

AWS EKS – Part 15 – Restrict Node IMDS access to Secure AWS account access

To install VPC CNI with IRSA, setup IRSA first from the above link, then run:

Create the trust relationship for the Kubernetes ServiceAccount:

Important: Don’t change system:serviceaccount:kube-system:aws-node.

cat <<EOF > irsa-trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::231144931069:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/E950239D8127DF45DE327DBCB6E2F280"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-west-2.amazonaws.com/id/E950239D8127DF45DE327DBCB6E2F280:aud": "sts.amazonaws.com",
          "oidc.eks.eu-west-2.amazonaws.com/id/E950239D8127DF45DE327DBCB6E2F280:sub": "system:serviceaccount:kube-system:aws-node"
        }
      }
    }
  ]
}
EOF

Create a new role and attach policy:

aws iam create-role \
  --role-name Kubedemy_EKS_IRSA_AWS_VPC_CNI \
  --assume-role-policy-document file://irsa-trust-policy.json \
  --tags Key=owner,Value=kubedemy

aws iam attach-role-policy \
  --role-name Kubedemy_EKS_IRSA_AWS_VPC_CNI \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

Detach AmazonEKS_CNI_Policy policy from node IAM role:

aws iam detach-role-policy \
  --role-name Kubedemy_EKS_Managed_Nodegroup_Role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

Install or update an existing VPC CNI addon:

aws eks create-addon \
  --cluster-name kubedemy \
  --addon-name vpc-cni \
  --addon-version v1.18.1-eksbuild.1 \
  --service-account-role-arn arn:aws:iam::231144931069:role/Kubedemy_EKS_IRSA_AWS_VPC_CNI \
  --resolve-conflicts OVERWRITE \
  --configuration-values file://vpc-cni-configuration.yaml \
  --tags owner=kubedemy

aws eks update-addon \
  --cluster-name kubedemy \
  --addon-name vpc-cni \
  --service-account-role-arn arn:aws:iam::231144931069:role/Kubedemy_EKS_IRSA_AWS_VPC_CNI

Step 3 – Customise AWS VPC CNI:

AWS VPC CNI has many customisation options; for example, you can enable the VPC CNI network policies feature, change MTU, change maximum ENIs per node, change Pod security group enforcing mode, disable/enable Prometheus metrics, enable/disable automatic subnet discovery, enable bandwidth plugin, etc. These changes can be done during addon installation/upgrade or by using ENIConfig if enabled.

To enable the VPC CNI network policy feature:

# Create addon configuration.
cat <<EOF > vpc-cni-configuration.yaml
tolerations:
  - operator: Exists
enableNetworkPolicy: "true"
nodeAgent:
  enablePolicyEventLogs: "true"
  enableCloudWatchLogs: "false"
  healthProbeBindAddr: "8163"
  metricsBindAddr: "8162"
EOF

aws eks update-addon \
  --cluster-name kubedemy \
  --addon-name vpc-cni \
  --service-account-role-arn arn:aws:iam::231144931069:role/Kubedemy_EKS_IRSA_AWS_VPC_CNI \
  --resolve-conflicts OVERWRITE \
  --configuration-values file://vpc-cni-configuration.yaml

Read more for more options in the VPC CNI GitHub repository.

Conclusion:

AWS VPC CNI is one the most comprehensive CNI plugins for Kubernetes. It provides dozens of features and allows you to customise what you need in your environment. In future articles, we will go deep into some advanced features and unleash its capabilities to implement different scenarios and overcome its limitations.

If you like this series of articles, please share them and write your thoughts as comments here. Your feedback encourages me to complete this massively planned program. Just share them and provide feedback. I’ll make you an AWS EKS black belt.

Follow my LinkedIn https://linkedin.com/in/ssbostan

Follow Kubedemy’s LinkedIn https://linkedin.com/company/kubedemy

Follow Kubedemy’s Telegram https://telegram.me/kubedemy