# \[T&C\] Load balancing Kubernetes traffic with Thunder ADC and TKC using IPIP Tunnels **URL:** \ **Category:** DevOps\ **Tags:** slb-adc, kubernetes, tkc, thunder-adc, k8s\ **Created:** [October 25, 2022, 7:34pm UTC](https://community.a10networks.com/t/t-c-load-balancing-kubernetes-traffic-with-thunder-adc-and-tkc-using-ipip-tunnels/752 "2022-10-25T19:34:00Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![siddharthaa](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@siddharthaa](https://community.a10networks.com/u/siddharthaa)\ **Post date:** [October 25, 2022, 7:34pm UTC](https://community.a10networks.com/t/t-c-load-balancing-kubernetes-traffic-with-thunder-adc-and-tkc-using-ipip-tunnels/752/1 "2022-10-25T19:34:00Z")
In an earlier [article](https://community.a10networks.com/discussion/16961/thunder-adc-with-thunder-kubernetes-connector-tkc-using-crds#latest), we saw how you can use the Thunder Kubernetes Connector (TKC) to dynamically configure the Thunder ADC for load-balancing traffic to a Kubernetes cluster. In that article, the Thunder device load-balanced traffic to the worker nodes. Starting from [TKC v1.11](https://community.a10networks.com/discussion/16961/thunder-adc-with-thunder-kubernetes-connector-tkc-using-crds#latest), the TKC can dynamically configure Thunder ADC to send traffic to the back-end Pods using IPIP tunnels. For this, you also need to have a Kubernetes cluster with a CNI plugin that supports such tunnels (e.g. Calico CNI network plugin). # **Setup** The K8s cluster consists of: Master: 172.16.1.12 - Node: 172.16.1.13 - Node: 172.16.1.14 The Pod network is 192.168.0.0/16. ![Screenshot 2022-10-25 120707.jpg](https://us1.discourse-cdn.com/flex002/uploads/a10_community/original/1X/e36b309b3afacb027c0b86c2b5e313449fb93c1d.jpeg) The K8s version is: ```bash $ kubectl version --short Client Version: v1.21.2 Server Version: v1.21.2 ``` In this setup, we already have the Calico CNI network plugin and calicoctl CLI tool installed : ```bash $ calicoctl version Client Version: v3.19.1 Git commit: 6fc0db96 Cluster Version: v3.19.1 Cluster Type: k8s,bgp,kubeadm,kdd ``` ### Enable HTTPS management access on the Thunder ADC interface The Thunder Kubernetes Connector (TKC) will run as a Pod within the K8s cluster and will communicate with Thunder ADC on interface ethernet 1. For this, enable management access on that interface of the Thunder ADC: ```bash enable-management service https ethernet 1 ``` The current configuration of the Thunder device is: ```bash !64-bit Advanced Core OS (ACOS) version 5.2.1-p5, build 114 (Jul-14-2022,05:11) ! harmony-controller telemetry log-rate 1750 ! ip dns primary 8.8.8.8 ! ip dns secondary 9.9.9.9 ! timezone America/Los_Angeles ! ntp server time.google.com ! glm use-mgmt-port glm enable-requests ! interface management ip address 10.64.4.32 255.255.255.0 ip control-apps-use-mgmt-port ip default-gateway 10.64.4.1 ! interface ethernet 1 enable ip address 172.16.1.10 255.255.255.0 ! interface ethernet 2 ! ! enable-management service https ethernet 1 ! ip route 0.0.0.0 /0 172.16.1.1 ! sflow setting local-collection ! sflow collector ip 127.0.0.1 6343 ! ! end !Current config commit point for partition 0 is 0 & config mode is classical-mode vThunder# ``` ### Install A10 CRDs Install A10’s CRDs (Custom Resource Definition) in the Kubernetes cluster: ```bash $ kubectl create -f https://a10networks.github.io/tkc-doc/crd/a10-crd-installation.yaml $ kubectl api-resources | grep a10 clientssls a10clissl tkc.a10networks.com/v1alpha1 true ClientSsl healthmonitors a10hm tkc.a10networks.com/v1 true HealthMonitor natpools a10natpool tkc.a10networks.com/v1 true NatPool serverssls a10srvssl tkc.a10networks.com/v1alpha1 true ServerSsl servicegroups a10sg tkc.a10networks.com/v1 true ServiceGroup templateciphers a10tc tkc.a10networks.com/v1alpha1 true TemplateCipher virtualports a10vport tkc.a10networks.com/v1 true VirtualPort virtualservers a10vip tkc.a10networks.com/v1alpha1 true VirtualServer ``` ### Add routes for the Thunder device LIF addresses Determine the tunnel interface addresses on K8s master and worker nodes. On K8s master: ```bash $ ifconfig tunl0 tunl0: flags=193 mtu 1480 inet 192.168.219.64 netmask 255.255.255.255 tunnel txqueuelen 1000 (IPIP Tunnel) RX packets 20 bytes 1468 (1.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 44 bytes 8168 (7.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ``` On K8s worker node node01: ```bash $ ifconfig tunl0 tunl0: flags=193 mtu 1480 inet 192.168.196.128 netmask 255.255.255.255 tunnel txqueuelen 1000 (IPIP Tunnel) RX packets 44 bytes 8168 (7.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 20 bytes 1468 (1.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ``` On K8 worker node node02: ```bash $ ifconfig tunl0 tunl0: flags=193 mtu 1480 inet 192.168.140.64 netmask 255.255.255.255 tunnel txqueuelen 1000 (IPIP Tunnel) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ``` The TKC will configure the last usable address of the /26 Pod subnet on each master and worker node as the LIF address on the Thunder device: - Pod subnet = 192.168.219.64/26: last usable IP = 192.168.219.126 - Pod subnet = 192.168.196.128/26: last usable IP = 192.168.196.190 - Pod subnet = 192.168.140.64/26: last usable IP = 192.168.140.126 On the K8s Master add routes for these Thunder LIF addresses via the tunl0 interface: ```bash $ sudo ip route add 192.168.219.126/32 dev tunl0 via 172.16.1.10 onlink $ sudo ip route add 192.168.196.190/32 dev tunl0 via 172.16.1.10 onlink $ sudo ip route add 192.168.140.126/32 dev tunl0 via 172.16.1.10 onlink $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.16.1.1 0.0.0.0 UG 100 0 0 eth0 172.16.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 192.168.140.64 172.16.1.14 255.255.255.192 UG 0 0 0 tunl0 192.168.140.126 172.16.1.10 255.255.255.255 UGH 0 0 0 tunl0 192.168.196.128 172.16.1.13 255.255.255.192 UG 0 0 0 tunl0 192.168.196.190 172.16.1.10 255.255.255.255 UGH 0 0 0 tunl0 192.168.219.64 0.0.0.0 255.255.255.192 U 0 0 0 * 192.168.219.74 0.0.0.0 255.255.255.255 UH 0 0 0 cali8be648026d3 192.168.219.75 0.0.0.0 255.255.255.255 UH 0 0 0 calic5e10800701 192.168.219.76 0.0.0.0 255.255.255.255 UH 0 0 0 cali1cd14967bcc 192.168.219.126 172.16.1.10 255.255.255.255 UGH 0 0 0 tunl0 ``` ### Configure Calico Get the current config of the Felix component of Calico using: ```bash $ calicoctl get felixConfig -o yaml > felixConfig.yaml ``` Currently, the config looks as follows: ```bash $ cat felixConfig.yaml apiVersion: projectcalico.org/v3 items: - apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: creationTimestamp: "2021-07-14T23:17:14Z" name: default resourceVersion: "778" uid: 57e45741-dbf2-45e4-ad2c-27c3e5d4d268 spec: bpfLogLevel: "" ipipEnabled: true logSeverityScreen: Info reportingInterval: 0s kind: FelixConfigurationList metadata: resourceVersion: "48327230" ``` In this, the “ipipEnabled” flag is currently set to true. Modify this “ipipEnabled” flag to false: ```bash $ cat felixConfig.yaml apiVersion: projectcalico.org/v3 items: - apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: creationTimestamp: "2021-07-14T23:17:14Z" name: default resourceVersion: "778" uid: 57e45741-dbf2-45e4-ad2c-27c3e5d4d268 spec: bpfLogLevel: "" ipipEnabled: false logSeverityScreen: Info reportingInterval: 0s kind: FelixConfigurationList metadata: resourceVersion: "48327230" ``` Now apply this modified setting: ```bash $ calicoctl apply -f felixConfig.yaml Successfully applied 1 'FelixConfiguration' resource(s) ``` # **Deploy a web app** In the K8s cluster, deploy a web service named web01, with the Service type of ClusterIP: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: webmap01 data: index.html: "

This is web service web01

" --- apiVersion: v1 kind: Service metadata: name: web01 spec: type: ClusterIP ports: - name: http-port protocol: TCP port: 8080 targetPort: 80 selector: app: web01 --- apiVersion: apps/v1 kind: Deployment metadata: name: web01 spec: replicas: 2 selector: matchLabels: app: web01 template: metadata: labels: app: web01 spec: volumes: - name: volmap configMap: name: webmap01 containers: - name: nginx image: nginx:1.13 ports: - containerPort: 80 volumeMounts: - name: volmap mountPath: /usr/share/nginx/html/ ``` Apply the config: ```bash $ kubectl apply -f web01-clusterip.yaml configmap/webmap01 created service/web01 created deployment.apps/web01 created ``` Check pods: ```bash $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web01-68c59f6f4c-nd247 1/1 Running 0 11s 192.168.140.112 node02 web01-68c59f6f4c-qdbtz 1/1 Running 0 11s 192.168.140.114 node02 ``` Check services: ```bash $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 460d web01 ClusterIP 10.109.161.206 8080/TCP 16m ``` # **Configure TKC** #### Create K8s Secret for Thunder device username/password Create a Secret object with base64 encoding of the default username/password for the Thunder ADC (admin/a10): ```yaml apiVersion: v1 kind: Secret metadata: name: a10-secret type: Opaque data: username: YWRtaW4= password: YTEw ``` Apply it: ```bash $ kubectl apply -f a10-secret.yaml secret/a10-secret created ``` #### Configure RBAC Deploy ServiceAccount, ClusterRole, and ClusterRoleBinding objects. This ServiceAccount will subsequently be applied to the TKC, thereby granting the TKC the required permissions to access the various resources within the K8s cluster. ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: a10-ingress namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: a10-ingress rules: - apiGroups: [""] resources: ["pods", "nodes", "services", "endpoints", "secrets", "configmaps"] verbs: ["get", "list", "watch"] - apiGroups: ["extensions", "networking.k8s.io"] resources: ["ingresses"] verbs: ["get", "list", "watch"] - apiGroups: ["tkc.a10networks.com"] resources: ["natpools", "healthmonitors", "virtualservers", "servicegroups", "serverssls", "virtualports", "templateciphers", "clientssls", "serversssls/status", "clientssls/status"] verbs: ["get", "watch", "list", "patch", "create", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: a10-ingress roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: a10-ingress subjects: - kind: ServiceAccount name: a10-ingress namespace: default ``` #### Deploy TKC Here we deploy TKC version v1.11.1.2. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: a10-thunder-kubernetes-connector spec: replicas: 1 selector: matchLabels: app: thunder-kubernetes-connector template: metadata: labels: app: thunder-kubernetes-connector spec: serviceAccountName: a10-ingress containers: - name: thunder-kubernetes-connector image: a10networks/a10-kubernetes-connector:1.11.1.2 imagePullPolicy: IfNotPresent env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: WATCH_NAMESPACE value: default - name: CONTROLLER_URL value: https://172.16.1.10 - name: ACOS_USERNAME_PASSWORD_SECRETNAME value: a10-secret - name: PARTITION value: shared - name: CONFIG_OVERLAY value: "true" - name: OVERLAY_ENDPOINT_IP value: 172.16.1.10 - name: VTEP_ENCAPSULATION value: ip-encap args: - --watch-namespace=$(WATCH_NAMESPACE) - --use-node-external-ip=true - --patch-to-update=true - --safe-acos-delete=true - --use-ingress-class-only=true - --ingress-class=a10-ext ``` Apply it: ```bash $ kubectl apply -f a10-tkc.yaml deployment.apps/a10-thunder-kubernetes-connector created $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES a10-thunder-kubernetes-connector-7ccd6ff787-bcp4t 1/1 Running 0 14m 192.168.140.111 node02 web01-68c59f6f4c-nd247 1/1 Running 0 17m 192.168.140.112 node02 web01-68c59f6f4c-qdbtz 1/1 Running 0 17m 192.168.140.114 node02 ``` In the above configuration, the following lines define the IPIP tunnel configuration: ```yaml - name: CONFIG_OVERLAY value: "true" - name: OVERLAY_ENDPOINT_IP value: 172.16.1.10 - name: VTEP_ENCAPSULATION value: ip-encap ``` Here the OVERLAY\_ENDPOINT\_IP is the physical IP of the Thunder device, in this case, 172.16.1.10 ### Deploy Ingress Resource Deploy an Ingress Resource with an ingress class of a10-ext (this should match the ingress-class value in the TKC manifest file) :a10-ext ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: a10-ingress01 namespace: default annotations: kubernetes.io/ingress.class: a10-ext spec: rules: - host: web01.a10tests.com http: paths: - path: / pathType: Prefix backend: service: name: web01 port: number: 8080 ``` Apply it: ```bash $ kubectl apply -f a10-ingress.yaml ingress.networking.k8s.io/a10-ingress01 created $ kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE a10-ingress01 web01.a10tests.com 80 10s ``` ### Deploy A10 CRDs for SLB objects _HealthMonitor Resource_ ```yaml apiVersion: "tkc.a10networks.com/v1" kind: HealthMonitor metadata: name: hm-http spec: name: hm-http type: http port: 80 retry: 5 interval: 10 timeout: 2 httpMethod: "GET" httpUrl: "/" httpCode: 200 ``` Apply it: ```bash $ kubectl apply -f crd-hm-http.yaml healthmonitor.tkc.a10networks.com/hm-http created $ kubectl get a10hm NAME NAME STATUS TYPE HOST URL CODE hm-http hm-http Active http / 200 ``` _ServiceGroup Resource_ Note that the ServiceGroup Resource binds to a specific Service Resource, in this case, web01: ```yaml apiVersion: "tkc.a10networks.com/v1" kind: ServiceGroup metadata: name: sg-web01 spec: name: "sg-web01" service: "web01" healthCheckDisable: 1 serverHealthCheck: "hm-http" ``` Apply it: ```bash $ kubectl apply -f crd-sg-web01.yaml servicegroup.tkc.a10networks.com/sg-web01 created $ kubectl get a10sg NAME NAME STATUS SERVICE PROTOCOL METHOD sg-web01 sg-web01 Active web01 tcp round-robin ``` _VirtualServer Resource_ ```yaml apiVersion: "tkc.a10networks.com/v1alpha1" kind: VirtualServer metadata: name: vip-web01 spec: name: vip-web01 ip-address: 172.16.1.251 ``` Apply it: ```bash $ kubectl apply -f crd-vip-web01.yaml virtualserver.tkc.a10networks.com/vip-web01 created $ kubectl get a10vip NAME NAME VIP IPAM STATUS vip-web01 vip-web01 172.16.1.251 Active ``` _VirtualPort Resource_ Note that the VirtualPort resource binds to a specific Ingress resource, in this case, a10-ingress01: ```yaml apiVersion: "tkc.a10networks.com/v1" kind: VirtualPort metadata: name: vport-web01-http spec: virtualServer: "vip-web01" port: 80 protocol: "http" ingress: "a10-ingress01" snatAuto: 1 ``` Apply it: ```bash $ kubectl apply -f crd-vport-web01-http.yaml virtualport.tkc.a10networks.com/vport-web01-http created $ kubectl get a10vport NAME PORT PROTOCOL STATUS VIRTUALSERVER INGRESS vport-web01-http 80 http Active vip-web01 a10-ingress01 ``` # **Verification** On the Thunder ADC, you will see the following configuration has been added by the TKC: ```auto interface lif acosk8s-140_64 ip address 192.168.140.126 255.255.255.192 ! interface lif acosk8s-196_128 ip address 192.168.196.190 255.255.255.192 ! interface lif acosk8s-219_64 ip address 192.168.219.126 255.255.255.192 ! ! health monitor hm-http retry 5 interval 10 timeout 2 method http port 80 expect response-code 200 url GET / ! slb server 192.168.140.112 192.168.140.112 health-check hm-http port 80 tcp ! slb server 192.168.140.114 192.168.140.114 health-check hm-http port 80 tcp ! slb service-group sg-web01 tcp health-check-disable member 192.168.140.112 80 member 192.168.140.114 80 ! slb template http default-a10-ingress01-httpTemplate url-switching starts-with / service-group sg-web01 ! slb virtual-server vip-web01 172.16.1.251 port 80 http source-nat auto service-group sg-web01 template http default-a10-ingress01-httpTemplate ! ! overlay-tunnel vtep 183 encap ip-encap local-ip-address 172.16.1.10 remote-ip-address 172.16.1.12 lif acosk8s-219_64 remote-ip-address 172.16.1.13 lif acosk8s-196_128 remote-ip-address 172.16.1.14 lif acosk8s-140_64 ! ``` As one can see, here the real servers are the Pod IP addresses. On the client machine, we have a DNS entry for “[web01.a10tests.com](http://web01.a10tests.com)” mapped to the VIP on the Thunder ADC. Now we can access the website [http://web01.a10tests.com](http://web01.a10tests.com) from this client machine: ![Screenshot 2022-10-10 210934.jpg](https://us1.discourse-cdn.com/flex002/uploads/a10_community/original/1X/ae6c4014e62f86ca0ba70c60cf6d05ac9ae0c2aa.jpeg) On the Thunder ADC, you can verify the SLB session entry: ```auto vThunder#sh session Prot Forward Source Forward Dest Reverse Source Reverse Dest Age Hash Flags Type --------------------------------------------------------------------------------------------------------------------------------------------------- Tcp 172.16.1.1:55268 172.16.1.251:80 192.168.140.112:80 192.168.140.126:24128 600 1 NSe1Cf0r0 SLB-L7 Total Sessions: 1 vThunder# ```