Hi.
We have SLB Virtual Service with type TCP-Proxy and we’ve also enabled Insert Client IP option in TCP-Proxy template but all traffic is forwarded from AX1030 internal IP .The question is how we can get source client IP ?
If you’re using a vport of type TCP, e.g., port 25 tcp, then you need to look in the TCP header for the information. Similar to an HTTP Template which inserts the client ip in the HTTP Header as a Header Field, a TCP Template mucks with the Options in the TCP Header. In my experiment, the client IP address was inserted as a TCP option just after the Window Scale option (plus a 0x01 pad). 1c 07 01 64 40 01 64 which corresponds with the IP address 100.64.1.100 1c - option type (not recognized by my wireshark) 07 - length (7 bytes) 01 - pad 64 40 01 64 - Client-IP in hex.
Thank you for your reply but the problem is how our proxy under AX1030 can define the source client IP when we have virtual service with type tcp-proxy ?
You need to find out what options your proxy supports.
If your proxy can not read the inserted Client-IP in the TCP-Options, then you need to use another way.
If your proxy supports reading the X-Forwarded-For HTTP header, you need to make sure the AX1030 inserts it.
Normally you would set you Virtual Service to use HTTP/HTTPS and inserting the headers is very easily done in the HTTP Template.
If for some reason you can not use HTTP, you could look at using aFlex to insert the header in the raw TCP data stream.
Is there a particular reason why you are using TCP-Proxy and not HTTP?
It seems that our proxy can’t read the inserted Client-IP in the TCP-Options or I can’t find the way how to do this,we use envoy for proxy under AX1030.Unfortunately we can’t use HTTP because we have a lot of connections to our service and in case we’ll use HTTP I think AX1030 can’t handle such amount of connections.Can you help with aFlex, how can I insert the header in the raw TCP data stream ? Maybe you have any example ?
Why do you think the A10 can not handle the nr of connections?
The use of aFlex is going to be much more heavy on your system than using HTTP.
The aFlex would have look at all packets and buffer data for every connection that needs to have te header inserted.
Actually it will need to add the header multiple times, every time a new request is coming in, even if it using an existing connection.
What does the Memory and Data CPU usage look like on peak moment?
If your device can not handle the extra processing required for HTTP, you should not use aFLex.
You could change your network setup, so that the return traffic from your proxy goes back to the A10… so you can remove source-NAT.
What other services are you running on the AX 1030? If you are also using it for a VIP with SSL-Offload, I would strongly suggest upgrading to a newer system as the SSL chipset on the AX is extremely outdated.
It’s difficult to say how many connections we have but I can say that we throughput 100k requests per second and we will have more.We don’t terminate SSL on AX1030 side,that deal our envoy proxy.Actually I thought about how to remove source-nat but for some reason I didn’t find the way how make our service works without source-nat.
Now when we use TCP type of our virtual service CPU and Memmory usage are very-very low.I can’t find any benchmarks for AX1030 to see how many HTTP connections it can handle and I decided to use TCP to hedge.
In an ideal situation the AX1030 should be able to do about 400K new TCP connections per second.
And depending on the re-use of these connections well over 1 million HTTP Requests per second. I expect it should be able to handle over 6 million concurrent connections.
But again that all depends on your setup.
If I were you, I would set the port type to HTTP and use the template, and then see what the performance impact is.
And if you expect to grow more than the unit can possible handle, get the Thunder 1040S 
Thank you for your suggestion but our hoster can only provide us AX1030 and these is the only option for us.
Maybe you can help how I can get rid of source-nat ? Maybe this somehow will help me ?
I actually already told you
You could change your network setup, so that the return traffic from your proxy goes back to the A10… so you can remove source-NAT.
Source-Nat is used to make sure the traffic from the A10 to the Proxy also comes back to the A10.
The only way to remove the need for Source-Nat is by making sure the Proxy sends the return traffic back to the A10.
So if you know the client IP’s, configure the proxy in such a way that for the client IP’s as destination it uses de A10 as next hop. If the Proxy and A10 are not part of the same broadcast network, you also need to fix the routing on the devices between the A10 and Proxy.
The tcp-proxy template and tcp-proxy vport handle insert-client-ip in the same way as the tcp template and vport: by inserting the client ip as a TCP Option in the TCP Header. As do how to read the TCP Option at your application server, I’ve heard of customers using firewall logging to record the value. I’m not sure what possibilities exist for the Application to read the value directly from the TCP socket (I took a seminar on UNIX tcp socket programming some 16 years ago but never used the knowledge). The vport type HTTP with an HTTP template with insert-client-ip sounds like it may be the way to go. It’s been optimized pretty well over the past decade. I agree with diederik that Source-NAT is largely a solution to a routing problem: how does one get the server’s reply packets back to the ADC in order to do whatever L4/L7 magic your need done without making the ADC the default gateway for all traffic from the server. There are other uses as well, but the return traffic challenge is the big one. If you can solve the problem with routing, then you don’t need insert-client-ip If the information so far isn’t enough to resolve the design/config challenge, you might be dealing with a situation perhaps too complex to handle in an online forum. Maybe contact A10 Professional Service? I would advise that especially if you wish to use the aFlex based solution.
Switched to HTTP and faced with another issue,so we can’t terminate SSL on ADC side cause we have smth like 1500 different certificates and the amount of certificates are growing every day,we terminate SSL on our proxy side.And as I understand we can’t bypass through ADC SSL traffic without termination, am I right ?
You have a loadbalancer in front of a proxy in front of websites you are hosting yourself?
I suppose the proxy is doing caching?
You can actually have the ADC bypass SSL traffic… but that means you still can not add the HTTP Header, as for HTTP Header insertion you need to alter the payload of the packets which are SSL encrypted.
Without knowing the total structure of your setup, I believe you only have 2 options. Either remove the need for Source-NAT by possibly moving the location of the A10 ADC or the proxy, as the proxy is doing SSL Termination the proxy is the most logical device to do HTTP Header updates and Source-NAT.
Or use insert-client-ip and find out how you can make your Proxy read the Client-IP’s from the TCP Options.
Like ericn suggested, you might want to get in touch with you A10 Sales/Systems Engineer so they kan have a look at your setup and possible help getting A10 Professional Services involved.
The scheme is like below:
ADC with external IP address and Source-NAT to the internal IP address → 2 Envoy proxies in Service Group that are balanced with the help of ADC by TCP type.
The only way that can help us is to remove the Source-NAT from this chain but how can we route traffic from external IP address to our internal IP addresses ?
You do not need source nat to be able to route traffic from external IP addressing to internal IP adressing. Basic routing is done on destination address, and as long as the ADC knows how to forward the traffic to your internal network (I suppose it has an interface in your internal network) it will work.
To forward traffic from the ADC to the servioce group, the destination address is altered. (Destination NAT). And that happens automatically when you have your ADC point the traffic to the IP addresses of the Envoy Proxy servers that are in your Service Group.
Lets say we use a client from the internet which has a source in the 200.x.y.z range.
And you service has a public IP address of 100.0.0.100
The VIP on the ADC is 10.0.0.100 and the Envoy server the traffic gets forwarded to has 192.168.0.100 a private address.
What you will see is that the external ip addressing is used for the outside session on the ADC and from the ADC without the Source-NAT it will only change the destination IP address.
From client on the internet to the ADC
external scr: 200.x.y.z
external dst: 100.0.0.100
From the ADC to the Envoy Proxy
internal scr: 200.x.y.z
internal dst: 192.168.0.100
So when the traffic needs to go back from the proxy to the client om the internet, it will use the 200.x.y.z as destination address.
Now you just need to make sure that that traffic passed from the Envoy Proxy through the A10 ADC back to the internet… when traffic returns over the A10 ADC, it will recognise the session and will Source-NAT it automatically back to the VIP address.
From the Envoy Proxy to the ADC
internal src: 192.168.0.100
internal dst: 200.x.y.z
From the ADC to the client on the internet
external src: 100.0.0.100
external dst: 200.x.y.z
Thank you for the great answer but I can’t make it works.I have VIP with real external address and when disable Source-NAT to my internal network Virtual Service immediately refuses to work.In my routing table on the ADX side I have only one route from 0.0.0.0 /0 to my default gateway.Don’t know what to try 
Ok, that is most likely because the Envoy proxies have a default gateway that is not the A10. Or maybe there is router between the A10 and the proxy?
You need to take a look at the proxy and see where it sends the traffic that needs to go back to the internet. Check it’s routing table.
If the Proxy and the A10 share the same broadcast domain, then you need to set the default gateway of the proxy to the A10 internal address. I assume you have VRRP-A setup, so set it to the floating address of your A10 cluster.
I assume this is the case as the A10 only has a default gateway set, and no specific routes for the internal network.
If there is one or more routers between the A10 and the proxy, you also need to make sure that those routers send the traffic in the right direction.
But, make sure that this does not break other traffic flows in your network.
Understand that if you do this, all traffic towards the internet from the proxy will go towards the A10.
So if there is a service on the proxy that needs to update things like signatures/URL Classification lists/or maybe needs to talk to other devices in your network that are not in a broadcast domain of one of the interfaces, you need to fix that as well. Maybe through specific routes or by enabling outbound NAT on the A10 for the proxies.
I would strongly suggest you contact the A10 Business partner or your A10 Systems Engineer/Account Team so they can assist you with finding out what the best approach is here.
Unfortunately we don’t have any support from A10 and this is pain.Our hosting company provide us AX1030 load balancer but it seems that they don’t know much about it’s configuration or they don’t want to do this.
Well, the routing part is not specific to A10.
The issue you are having getting the traffic back from the proxy to the A10, would be the same if you had any other equipment in place of the A10.
This is basic networking.
All they have to do is make sure the traffic from the Proxy is send back to the A10.
So the configuration of the proxy needs to change first… not the A10.
But of course you can always contact A10 to see who the contacts are in your region.
You could hire A10’s Professional Services team to have a look.