[T&C] Thunder ADC with Thunder Kubernetes Connector (TKC) using CRDs
siddharthaa
Member, Administrator admin
In an earlier article, we saw how you can use the Thunder Kubernetes Connector (TKC) to dynamically configure the Thunder ADC for load-balancing traffic to a Kubernetes cluster. In that article, we specified the SLB configuration using annotations in an Ingress resource.
Starting from TKC v1.11, Thunder Kubernetes Connector (TKC) allows you to define your own Custom Resource Definition (CRD) for SLB objects in the Kubernetes API. These SLB objects are then configured with custom options on the Thunder device.
TKC supports Custom Resource Definition (CRD) for the following SLB objects:
- virtual-server
- virtual-port
- service-group
- slb cipher template
- slb server-ssl template
- slb client-ssl template
- health monitor
- ip nat pool
In this article, we will see how you can specify the SLB configuration using CRDs.
Video Demo
For a demo of the below steps, see:
Setup
The K8s cluster consists of:
- Master: 172.16.1.12
- Node: 172.16.1.13
- Node: 172.16.1.14
The Pod network is 192.168.0.0/16.
The K8s version is:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
The current configuration of the Thunder device is:
harmony-controller telemetry log-rate 1750 ! ip dns primary 8.8.8.8 ! ip dns secondary 9.9.9.9 ! timezone America/Los_Angeles ! ntp server time.google.com ! glm use-mgmt-port glm enable-requests ! interface management ip address 10.64.4.32 255.255.255.0 ip control-apps-use-mgmt-port ip default-gateway 10.64.4.1 enable ! interface ethernet 1 enable ip address 172.16.1.10 255.255.255.0 ! interface ethernet 2 ! ! ip route 0.0.0.0 /0 172.16.1.1 ! sflow setting local-collection ! sflow collector ip 127.0.0.1 6343 ! ! end
Configuration
Deploy a web app in the K8s cluster
In the K8s cluster, deploy a web service named web01, with Service type as NodePort:
$ cat web01.yaml apiVersion: v1 kind: ConfigMap metadata: name: webmap01 data: index.html: "<html><h1>This is web service web01</h1><html>" --- apiVersion: v1 kind: Service metadata: name: web01 spec: type: NodePort ports: - name: http-port protocol: TCP port: 8080 targetPort: 80 selector: app: web01 --- apiVersion: apps/v1 kind: Deployment metadata: name: web01 spec: replicas: 3 selector: matchLabels: app: web01 template: metadata: labels: app: web01 spec: volumes: - name: volmap configMap: name: webmap01 containers: - name: nginx image: nginx:1.13 ports: - containerPort: 80 volumeMounts: - name: volmap mountPath: /usr/share/nginx/html/ $ kubectl apply -f web01.yaml $ kubectl get pods NAME READY STATUS RESTARTS AGE web01-68c59f6f4c-dcvs2 1/1 Running 0 6d web01-68c59f6f4c-qrfjd 1/1 Running 0 6d web01-68c59f6f4c-z865r 1/1 Running 0 6d
Enable HTTPS management access on the Thunder ADC interface
The Thunder Kubernetes Connector (TKC) will run as a Pod within the K8s cluster and will communicate with Thunder ADC on interface ethernet 1. For this, enable management access on that interface of the Thunder ADC:
enable-management service https ethernet 1
Deploy K8s Resources
Create K8s Secret for Thunder device username/password
Create a Secret object with base64 encoding of the default username/password for the Thunder ADC (admin/a10):
$ cat a10-secret.yaml apiVersion: v1 kind: Secret metadata: name: a10-secret type: Opaque data: username: YWRtaW4= password: YTEw $ kubectl apply -f a10-secret.yaml secret/a10-secret created
Create ServiceAccount, ClusterRole, and ClusterRoleBinding objects
Deploy ServiceAccount, ClusterRole, and ClusterRoleBinding objects. This ServiceAccount will subsequently be applied to the TKC, thereby granting the TKC the required permissions to access the various resources within the K8s cluster.
$ cat a10-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: a10-ingress namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: a10-ingress rules: - apiGroups: [""] resources: ["pods", "nodes", "services", "endpoints", "secrets", "configmaps"] verbs: ["get", "list", "watch"] - apiGroups: ["extensions", "networking.k8s.io"] resources: ["ingresses"] verbs: ["get", "list", "watch"] - apiGroups: ["tkc.a10networks.com"] resources: ["natpools", "healthmonitors", "virtualservers", "servicegroups", "serverssls", "virtualports", "templateciphers", "clientssls", "serversssls/status", "clientssls/status"] verbs: ["get", "watch", "list", "patch", "create", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: a10-ingress roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: a10-ingress subjects: - kind: ServiceAccount name: a10-ingress namespace: default $ kubectl apply -f a10-rbac.yaml serviceaccount/a10-ingress created clusterrole.rbac.authorization.k8s.io/a10-ingress created clusterrolebinding.rbac.authorization.k8s.io/a10-ingress created
Deploy Ingress Resource
Deploy an Ingress Resource with ingress class of a10-ext:
$ cat a10-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: a10-ingress01 namespace: default annotations: kubernetes.io/ingress.class: a10-ext spec: rules: - host: web01.a10tests.com http: paths: - path: / pathType: Prefix backend: service: name: web01 port: number: 8080 $ kubectl apply -f a10-ingress.yaml ingress.networking.k8s.io/a10-ingress01 created $ kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE a10-ingress01 <none> web01.a10tests.com 80 19s
Deploy TKC
Here we deploy TKC version v1.11.0.0.
$ cat a10-tkc.yaml apiVersion: apps/v1 kind: Deployment metadata: name: a10-thunder-kubernetes-connector spec: replicas: 1 selector: matchLabels: app: thunder-kubernetes-connector template: metadata: labels: app: thunder-kubernetes-connector spec: serviceAccountName: a10-ingress containers: - name: thunder-kubernetes-connector image: a10networks/a10-kubernetes-connector:1.11.0.0 imagePullPolicy: IfNotPresent env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: WATCH_NAMESPACE value: default - name: CONTROLLER_URL value: https://172.16.1.10 - name: ACOS_USERNAME_PASSWORD_SECRETNAME value: a10-secret - name: PARTITION value: shared args: - --watch-namespace=$(WATCH_NAMESPACE) - --use-node-external-ip=true - --patch-to-update=true - --safe-acos-delete=true - --use-ingress-class-only=true - --ingress-class=a10-ext $ kubectl apply -f a10-tkc.yaml deployment.apps/a10-thunder-kubernetes-connector created
Install and Deploy CRDs for SLB objects
Install TKC CRDs
Install A10's TKC CRDs in the Kubernetes cluster:
$ kubectl create -f https://a10networks.github.io/tkc-doc/crd/a10-crd-installation.yaml $ kubectl api-resources | grep a10 clientssls a10clissl tkc.a10networks.com/v1alpha1 true ClientSsl healthmonitors a10hm tkc.a10networks.com/v1 true HealthMonitor natpools a10natpool tkc.a10networks.com/v1 true NatPool serverssls a10srvssl tkc.a10networks.com/v1alpha1 true ServerSsl servicegroups a10sg tkc.a10networks.com/v1 true ServiceGroup templateciphers a10tc tkc.a10networks.com/v1alpha1 true TemplateCipher virtualports a10vport tkc.a10networks.com/v1 true VirtualPort virtualservers a10vip tkc.a10networks.com/v1alpha1 true VirtualServer
HealthMonitor Resource
$ cat crd-hm-http.yaml apiVersion: "tkc.a10networks.com/v1" kind: HealthMonitor metadata: name: hm-http spec: name: hm-http type: http port: 80 retry: 5 interval: 10 timeout: 2 httpMethod: "GET" httpUrl: "/" httpCode: 200 [a10tme@master ~]$ kubectl apply -f crd-hm-http.yaml healthmonitor.tkc.a10networks.com/hm-http created [a10tme@master ~]$ kubectl get a10hm NAME NAME STATUS TYPE HOST URL CODE hm-http hm-http Active http / 200
ServiceGroup Resource
Note that the ServiceGroup Resource binds to a specific Service Resource, in this case, web01:
$ cat crd-sg-web01.yaml apiVersion: "tkc.a10networks.com/v1" kind: ServiceGroup metadata: name: sg-web01 spec: name: "sg-web01" service: "web01" healthCheck: "hm-http" $ kubectl apply -f crd-sg-web01.yaml servicegroup.tkc.a10networks.com/sg-web01 created $ kubectl get a10sg NAME NAME STATUS SERVICE PROTOCOL METHOD sg-web01 sg-web01 Active web01 tcp round-robin
VirtualServer Resource
$ cat crd-vip-web01.yaml apiVersion: "tkc.a10networks.com/v1alpha1" kind: VirtualServer metadata: name: vip-web01 spec: name: vip-web01 ip-address: 172.16.1.251 $ kubectl apply -f crd-vip-web01.yaml virtualserver.tkc.a10networks.com/vip-web01 created $ kubectl get a10vip NAME NAME VIP STATUS vip-web01 vip-web01 172.16.1.251 Active
VirtualPort Resource
Note that the VirtualPort resource binds to a specific Ingress resource, in this case, a10-ingress01:
$ cat crd-vport-web01-http.yaml apiVersion: "tkc.a10networks.com/v1" kind: VirtualPort metadata: name: vport-web01-http spec: virtualServer: "vip-web01" port: 80 protocol: "http" ingress: "a10-ingress01" snatAuto: 1 $ kubectl apply -f crd-vport-web01-http.yaml virtualport.tkc.a10networks.com/vport-web01-http created $ kubectl get a10vport NAME PORT PROTOCOL STATUS VIRTUALSERVER INGRESS ROUTE vport-web01-http 80 http Active vip-web01 a10-ingress01
Verification
On the Thunder ADC, you will see the following SLB configuration has been added by the TKC:
health monitor hm-http retry 5 interval 10 timeout 2 method http port 80 expect response-code 200 url GET / ! slb server 172.16.1.13 172.16.1.13 port 30451 tcp ! slb server 172.16.1.14 172.16.1.14 port 30451 tcp ! slb service-group sg-web01 tcp health-check hm-http member 172.16.1.13 30451 member 172.16.1.14 30451 ! slb template http default-a10-ingress01-httpTemplate url-switching starts-with / service-group sg-web01 ! slb virtual-server vip-web01 172.16.1.251 port 80 http source-nat auto service-group sg-web01 template http default-a10-ingress01-httpTemplate
On the client machine, we have a DNS entry for "web01.a10tests.com" mapped to the VIP on the Thunder ADC. Now we can access the website http://web01.a10tests.com from this client machine:
$ curl http://web01.a10tests.com <html><h1>This is web service web01</h1><html>