Contact us

AWS EKS – Part 29 – Setup AWS Load Balancer Controller

Saeid Bostandoust

AWS EKS supports LoadBalancer services to expose your applications outside the clusters. By default, when you create a LoadBalancer Service, AWS creates a Classic Load Balancer(CLB), which is the previous generation and almost a deprecated load balancer in AWS services. On the other hand, we have a Kubernetes SIG project called AWS Load Balancer Controller, which replaces the CLB with the Network Load Balancer(NLB), which provides a high-performance load balancer with various features, and we will introduce it in this lesson. In addition, the AWS Load Balancer Controller can create Application Load Balancers, which can be used as an Ingress Controller to expose Ingress resources in Kubernetes, which will be introduced in the next lesson.

Follow our social media:

https://linkedin.com/in/ssbostan

https://linkedin.com/company/kubedemy

https://youtube.com/@kubedemy

https://telegram.me/kubedemy

Register for the FREE EKS Tutorial:

If you want to access the course materials, register from the following link:

Register for the FREE AWS EKS Black Belt Course

Kubernetes Service Types:

The Kubernetes Service resource supports four different types.

  • ClusterIP: This service type assigns an IP address within the Service CIDR block, “10.100.0.0/16 or 172.20.0.0/16 in EKS”, to access the selected Pods from within the cluster. It provides a way to connect to Pods using a permanent IP address and a DNS name, as Pods and their IP addresses are ephemeral.
  • NodePort: This service type provides all the ClusterIP features and opens a specific port on worker nodes. Traffic sent to that port number on the worker nodes is forwarded to the ClusterIP and then to the selected Pods. This service type allows you to expose the application outside the cluster.
  • LoadBalancer: This service type provides all the NodePort features in addition to creating an external Load Balancer to put all worker nodes listening to that specific port behind the load balancer to automatically distribute the traffic between different nodes. Creating external Load Balancers requires additional controllers inside the Kubernetes clusters. This service type is not generally available on Bare-metal Kubernetes clusters, “you can use MetalLB, OpenELB, Cilium, etc.” and in cloud environments, cloud-controller-manager is responsible for creating external load balancers with the cloud. The AWS Load Balancer Controller works in this layer to create NLB within our AWS account and send traffic to Kubernetes Pods.
  • ExternalName: Unlike other types, this type does not expose Kubernetes Pods in/out of the cluster. Instead, it is used to introduce external applications within the cluster and allows Pods to access external entities using DNS names.

Classic Load Balancer vs. Network Load Balancer:

CLB provides both Layer 4 and Layer 7 load balancers at the same time, and it supports some Layer 7 features like routing, health checks, and sticky sessions, but its performance is much lower than NLB due to additional processing for Layer 7 features even if you don’t use them. When you scale a CLB load balancer, it takes time to change its traffic distributing pattern. NLB supports Layer 4 only and provides all CLB features except the Layer 7 ones. It gives a really high performance, and it scales almost instantly. If Layer 7 features are needed, use an Application Load Balancer(ALB) instead.

CLB Preview before NLB installation:

When you expose your application in Kubernetes using the LoadBalancer service, AWS creates a Classic Load Balancer and provides you with a sub-domain to access the service. All traffic to that URL will be forwarded to the CLB and then to a NodePort in worker nodes, and from there, Kubernetes is responsible for forwarding traffic to the selected Pods. Working with CLB doesn’t need additional IAM policies; all required policies are already bundled into the Kubernetes Cluster IAM Role.

AWS Load Balancer Controller setup procedure:

  • Create an IAM policy for the AWS Load Balancer Controller.
  • Assign the IAM policy to the node IAM role or IRSA.
  • Tag VPC Subnets used by AWS Load Balancer Controller.
  • Setup AWS Load Balancer Controller using Helm chart.
  • Test LoadBalancer and check Network Load Balancer.

Step 1 – Create IAM Policy for AWS LBC Service:

To allow the AWS Load Balancer Controller to create and manage Load Balancers within our AWS account, we need to create the following policy:

cat <<EOF > aws-load-balancer-controller-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeTags",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeCoipPools",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeTrustStores"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "waf-regional:GetWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:AssociateWebACL",
                "waf-regional:DisassociateWebACL",
                "wafv2:GetWebACL",
                "wafv2:GetWebACLForResource",
                "wafv2:AssociateWebACL",
                "wafv2:DisassociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:CreateProtection",
                "shield:DeleteProtection"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSecurityGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:DeleteRule"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:DeleteTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule"
            ],
            "Resource": "*"
        }
    ]
}
EOF

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerPolicy \
    --policy-document file://aws-load-balancer-controller-policy.json

Step 2 – Assign LBC IAM Policy to Node / IRSA:

The AWS Load Balancer Controller will use the created IAM policy. The best practice is to setup IRSA for the cluster, assign that policy to an IAM role and use it for the LBC service account. We can also attach the policy to the node IAM role.

AWS EKS – Part 13 – Setup IAM Roles for Service Accounts (IRSA)

AWS EKS – Part 14 – Setup EKS Pod Identities to access AWS resources

AWS EKS – Part 15 – Restrict Node IMDS access to Secure AWS account access

To attach the policy to the node IAM role:

aws iam attach-role-policy \
  --policy-arn arn:aws:iam::231144931069:policy/AWSLoadBalancerControllerPolicy \
  --role-name Kubedemy_EKS_Managed_Nodegroup_Role

Step 3 – Tag VPC Subnets for AWS LBC:

The AWS Load Balancer Controller auto-discovers the subnets for creating NLBs and ALBs. One subnet is needed for a Network Load Balancer, and at least two subnets are required for an Application Load Balancer. Each subnet must have at least eight free IP addresses. It looks for one available subnet per availability zone; if more than one subnet matches the requirements, the first one will be selected based on their IDs in order.

In Public subnets, add the following tags:

kubernetes.io/role/elb=1
kubernetes.io/cluster/CLUSTER_NAME=shared

In Private subnets, add the following tags:

kubernetes.io/role/internal-elb=1
kubernetes.io/cluster/CLUSTER_NAME=shared

I want to add them to our public subnets:

aws ec2 create-tags \
  --resources subnet-0ff015478090c2174 subnet-01b107cea804fdff1 subnet-09b7d720aca170608 \
  --tags Key=kubernetes.io/role/elb,Value=1

aws ec2 create-tags \
  --resources subnet-0ff015478090c2174 subnet-01b107cea804fdff1 subnet-09b7d720aca170608 \
  --tags Key=kubernetes.io/cluster/kubedemy,Value=shared

Step 4 – Install AWS Load Balancer Controller:

To install the AWS Load Balancer Controller using Helm charts, use the following commands. The chart provides many customisation options, which you can find here. You can also install cert-manager operator before this chart and use it to generate certificates. If prometheus operator is installed, you can configure the helm chart to create Prometheus-related resources to scrape the LBC metrics.

helm repo add eks https://aws.github.io/eks-charts

helm repo update eks

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=kubedemy \
  --set serviceAccount.create=false \
  --set serviceAccount.name=default \
  --set defaultTargetType=ip \
  --namespace aws-load-balancer-controller \
  --create-namespace

Note: The cluster name should be the same as the cluster tag on subnets.

Note: If IRSA is used, create a ServiceAccount and provide Role ARN to the resource.

Note: If using any CNI other than VPC CNI, TargetType must be instance.

Step 5 – Test LBC and create Network Load Balancer:

To expose an application and create an external internet-facing Network Load Balancer, use the following annotations while creating the Service resource:

service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

Other annotations are explained here in the official documentation.

It should be something like this:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
    - port: 80

To confirm that, check the load balancer domain and in the AWS account:

Conclusion:

AWS Load Balancer Controller(LBC) is a way to create NLB and ALB load balancers, which replace the CLB load balancer. For Service resources, it mutates the resource using its mutating webhook and changes the loadBalancerClass option to service.k8s.aws/nlb and creates a Network Load Balancer for the service. Furthermore, when using VPC CNI + IP Target, the traffic will forwarded directly to the Pods, which makes more performance than other methods.

If you like this series of articles, please share them and write your thoughts as comments here. Your feedback encourages me to complete this massively planned program. Just share them and provide feedback. I’ll make you an AWS EKS black belt.

Follow my LinkedIn https://linkedin.com/in/ssbostan

Follow Kubedemy’s LinkedIn https://linkedin.com/company/kubedemy

Follow Kubedemy’s Telegram https://telegram.me/kubedemy

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

We use cookies to improve your experience on our website