Skip to main content
IntroFundamentalsObservabilitySecurityNetworkingAutomationAI/MLTroubleshooting
Switch to Essentials →

IP mode

As mentioned previously, the NLB we have created is operating in "instance mode". Instance target mode supports pods running on AWS EC2 instances. In this mode, AWS NLB sends traffic to the instances and the kube-proxy on the individual worker nodes forward it to the pods through one or more worker nodes in the Kubernetes cluster.

The AWS Load Balancer Controller also supports creating NLBs operating in "IP mode". In this mode, the AWS NLB sends traffic directly to the Kubernetes pods behind the service, eliminating the need for an extra network hop through the worker nodes in the Kubernetes cluster. IP target mode supports pods running on both AWS EC2 instances and AWS Fargate.

A side-by-side comparison of NLB target modes: in instance mode the target group registers EC2 instances on the NodePort and kube-proxy relays the connection to the ui Pod, while in IP mode the target group registers Pod IPs and the load balancer connects to the ui Pod directly

The diagram above compares the two modes side by side. The Service, the Network Load Balancer and its TCP listener on port 80 are identical in both — the only thing that changes is what gets registered in the load balancer's target group.

In instance mode the target group registers the EC2 worker nodes on the NodePort that Kubernetes allocated for the Service. A connection therefore arrives at a node first, and kube-proxy on that node relays it to a ui Pod. If the node that received the connection is not running a ui Pod, kube-proxy forwards it on to a node that is, adding a second network hop.

In IP mode the target group registers the Pod IP addresses instead, so the load balancer opens the connection straight to a ui Pod and kube-proxy is not in the data path at all. This works because the Amazon VPC CNI gives every Pod a first-class, routable VPC IP address.

There are several reasons why we might want to configure the NLB to operate in IP target mode:

  1. It creates a more efficient network path for inbound connections, bypassing kube-proxy on the EC2 worker node
  2. It removes the need to consider aspects such as externalTrafficPolicy and the trade-offs of its various configuration options
  3. An application is running on Fargate instead of EC2

Re-configuring the NLB​

Let's reconfigure our NLB to use IP mode and look at the effect it has on the infrastructure.

This is the patch we'll be applying to re-configure the Service:

~/environment/eks-workshop/modules/exposing/load-balancer/ip-mode/nlb.yaml
apiVersion: v1
kind: Service
metadata:
  name: ui-nlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
  namespace: ui

Apply the manifest with kustomize:

~$kubectl kustomize ~/environment/eks-workshop/modules/exposing/load-balancer/ip-mode | envsubst | kubectl apply -f -


It will take a few minutes for the configuration of the load balancer to be updated. Run the following command to ensure the annotation is updated:


~$kubectl describe service/ui-nlb -n ui
...
Annotations:              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
...


You should be able to access the application using the same URL as before, with the NLB now using IP mode to expose your application.


~$NLB_ARN=$(aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-ui-uinlb`) == `true`].LoadBalancerArn' | jq -r '.[0]')
~$TARGET_GROUP_ARN=$(aws elbv2 describe-target-groups --load-balancer-arn $NLB_ARN | jq -r '.TargetGroups[0].TargetGroupArn')
~$aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN
{
    "TargetHealthDescriptions": [
        {
            "Target": {
                "Id": "10.42.180.183",
                "Port": 8080,
                "AvailabilityZone": "us-west-2a"
            },
            "HealthCheckPort": "8080",
            "TargetHealth": {
                "State": "initial",
                "Reason": "Elb.RegistrationInProgress",
                "Description": "Target registration is in progress"
            }
        }
    ]
}


Notice that we've gone from the 3 targets we observed in the previous section to just a single target. Why is that? Instead of registering the EC2 instances in our EKS cluster the load balancer controller is now registering individual Pods and sending traffic directly, taking advantage of the AWS VPC CNI and the fact that Pods each have a first-class VPC IP address.


Let's scale up the ui component to 3 replicas see what happens:


~$kubectl scale -n ui deployment/ui --replicas=3
~$kubectl wait --for=condition=Ready pod -n ui -l app.kubernetes.io/name=ui --timeout=60s


Now check the load balancer targets again:


~$NLB_ARN=$(aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-ui-uinlb`) == `true`].LoadBalancerArn' | jq -r '.[0]')
~$TARGET_GROUP_ARN=$(aws elbv2 describe-target-groups --load-balancer-arn $NLB_ARN | jq -r '.TargetGroups[0].TargetGroupArn')
~$aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN
{
    "TargetHealthDescriptions": [
        {
            "Target": {
                "Id": "10.42.180.181",
                "Port": 8080,
                "AvailabilityZone": "us-west-2c"
            },
            "HealthCheckPort": "8080",
            "TargetHealth": {
                "State": "initial",
                "Reason": "Elb.RegistrationInProgress",
                "Description": "Target registration is in progress"
            }
        },
        {
            "Target": {
                "Id": "10.42.140.129",
                "Port": 8080,
                "AvailabilityZone": "us-west-2a"
            },
            "HealthCheckPort": "8080",
            "TargetHealth": {
                "State": "healthy"
            }
        },
        {
            "Target": {
                "Id": "10.42.105.38",
                "Port": 8080,
                "AvailabilityZone": "us-west-2a"
            },
            "HealthCheckPort": "8080",
            "TargetHealth": {
                "State": "initial",
                "Reason": "Elb.RegistrationInProgress",
                "Description": "Target registration is in progress"
            }
        }
    ]
}


As expected we now have 3 targets, matching the number of replicas in the ui Deployment.


If you want to wait to make sure the application still functions the same, run the following command. Otherwise you can proceed to the next module.


~$curl --head -X GET --retry 30 --retry-all-errors --retry-delay 15 --connect-timeout 30 --max-time 60 \
  -k $(kubectl get service -n ui ui-nlb -o jsonpath="{.status.loadBalancer.ingress[*].hostname}")