TCP-proxy Client-IP

Hi.
We have SLB Virtual Service with type TCP-Proxy and we've also enabled Insert Client IP option in TCP-Proxy template but all traffic is forwarded from AX1030 internal IP .The question is how we can get source client IP ?
Tagged:

Comments

  • ericnericn Member
    edited August 2018
    If you're using a vport of type TCP, e.g., port 25 tcp, then you need to look in the TCP header for the information. Similar to an HTTP Template which inserts the client ip in the HTTP Header as a Header Field, a TCP Template mucks with the Options in the TCP Header.

    In my experiment, the client IP address was inserted as a TCP option just after the Window Scale option (plus a 0x01 pad).

    1c 07 01 64 40 01 64 which corresponds with the IP address 100.64.1.100
    1c - option type (not recognized by my wireshark)
    07 - length (7 bytes)
    01 - pad
    64 40 01 64 - Client-IP in hex.
  • flaneurflaneur Member
    edited August 2018
    Thank you for your reply but the problem is how our proxy under AX1030 can define the source client IP when we have virtual service with type tcp-proxy ?
  • diederikdiederik Member
    edited August 2018
    You need to find out what options your proxy supports.

    If your proxy can not read the inserted Client-IP in the TCP-Options, then you need to use another way.
    If your proxy supports reading the X-Forwarded-For HTTP header, you need to make sure the AX1030 inserts it.

    Normally you would set you Virtual Service to use HTTP/HTTPS and inserting the headers is very easily done in the HTTP Template.

    If for some reason you can not use HTTP, you could look at using aFlex to insert the header in the raw TCP data stream.

    Is there a particular reason why you are using TCP-Proxy and not HTTP?
  • flaneurflaneur Member
    edited August 2018
    It seems that our proxy can't read the inserted Client-IP in the TCP-Options or I can't find the way how to do this,we use envoy for proxy under AX1030.Unfortunately we can't use HTTP because we have a lot of connections to our service and in case we'll use HTTP I think AX1030 can't handle such amount of connections.Can you help with aFlex, how can I insert the header in the raw TCP data stream ? Maybe you have any example ?
  • diederikdiederik Member
    edited August 2018
    Why do you think the A10 can not handle the nr of connections?

    The use of aFlex is going to be much more heavy on your system than using HTTP.
    The aFlex would have look at all packets and buffer data for every connection that needs to have te header inserted.
    Actually it will need to add the header multiple times, every time a new request is coming in, even if it using an existing connection.

    What does the Memory and Data CPU usage look like on peak moment?
    If your device can not handle the extra processing required for HTTP, you should not use aFLex.

    You could change your network setup, so that the return traffic from your proxy goes back to the A10... so you can remove source-NAT.

    What other services are you running on the AX 1030? If you are also using it for a VIP with SSL-Offload, I would strongly suggest upgrading to a newer system as the SSL chipset on the AX is extremely outdated.
  • flaneurflaneur Member
    edited August 2018
    It's difficult to say how many connections we have but I can say that we throughput 100k requests per second and we will have more.We don't terminate SSL on AX1030 side,that deal our envoy proxy.Actually I thought about how to remove source-nat but for some reason I didn't find the way how make our service works without source-nat.
    Now when we use TCP type of our virtual service CPU and Memmory usage are very-very low.I can't find any benchmarks for AX1030 to see how many HTTP connections it can handle and I decided to use TCP to hedge.
  • diederikdiederik Member
    edited August 2018
    In an ideal situation the AX1030 should be able to do about 400K new TCP connections per second.
    And depending on the re-use of these connections well over 1 million HTTP Requests per second. I expect it should be able to handle over 6 million concurrent connections.
    But again that all depends on your setup.

    If I were you, I would set the port type to HTTP and use the template, and then see what the performance impact is.

    And if you expect to grow more than the unit can possible handle, get the Thunder 1040S :)
  • flaneurflaneur Member
    edited August 2018
    Thank you for your suggestion but our hoster can only provide us AX1030 and these is the only option for us.
  • flaneurflaneur Member
    edited August 2018
    Maybe you can help how I can get rid of source-nat ? Maybe this somehow will help me ?
  • diederikdiederik Member
    edited August 2018
    I actually already told you :)
    You could change your network setup, so that the return traffic from your proxy goes back to the A10… so you can remove source-NAT.
    Source-Nat is used to make sure the traffic from the A10 to the Proxy also comes back to the A10.

    The only way to remove the need for Source-Nat is by making sure the Proxy sends the return traffic back to the A10.

    So if you know the client IP's, configure the proxy in such a way that for the client IP's as destination it uses de A10 as next hop. If the Proxy and A10 are not part of the same broadcast network, you also need to fix the routing on the devices between the A10 and Proxy.
  • ericnericn Member
    edited August 2018
    The tcp-proxy template and tcp-proxy vport handle insert-client-ip in the same way as the tcp template and vport: by inserting the client ip as a TCP Option in the TCP Header.

    As do how to read the TCP Option at your application server, I've heard of customers using firewall logging to record the value. I'm not sure what possibilities exist for the Application to read the value directly from the TCP socket (I took a seminar on UNIX tcp socket programming some 16 years ago but never used the knowledge).

    The vport type HTTP with an HTTP template with insert-client-ip sounds like it may be the way to go. It's been optimized pretty well over the past decade.

    I agree with diederik that Source-NAT is largely a solution to a routing problem: how does one get the server's reply packets back to the ADC in order to do whatever L4/L7 magic your need done without making the ADC the default gateway for all traffic from the server. There are other uses as well, but the return traffic challenge is the big one. If you can solve the problem with routing, then you don't need insert-client-ip

    If the information so far isn't enough to resolve the design/config challenge, you might be dealing with a situation perhaps too complex to handle in an online forum. Maybe contact A10 Professional Service? I would advise that especially if you wish to use the aFlex based solution.
  • flaneurflaneur Member
    edited September 2018
    Switched to HTTP and faced with another issue,so we can't terminate SSL on ADC side cause we have smth like 1500 different certificates and the amount of certificates are growing every day,we terminate SSL on our proxy side.And as I understand we can't bypass through ADC SSL traffic without termination, am I right ?
  • diederikdiederik Member
    edited September 2018
    You have a loadbalancer in front of a proxy in front of websites you are hosting yourself?
    I suppose the proxy is doing caching?

    You can actually have the ADC bypass SSL traffic... but that means you still can not add the HTTP Header, as for HTTP Header insertion you need to alter the payload of the packets which are SSL encrypted.

    Without knowing the total structure of your setup, I believe you only have 2 options.
    Either remove the need for Source-NAT by possibly moving the location of the A10 ADC or the proxy, as the proxy is doing SSL Termination the proxy is the most logical device to do HTTP Header updates and Source-NAT.

    Or use insert-client-ip and find out how you can make your Proxy read the Client-IP's from the TCP Options.

    Like ericn suggested, you might want to get in touch with you A10 Sales/Systems Engineer so they kan have a look at your setup and possible help getting A10 Professional Services involved.
  • flaneurflaneur Member
    edited September 2018
    The scheme is like below:
    ADC with external IP address and Source-NAT to the internal IP address -> 2 Envoy proxies in Service Group that are balanced with the help of ADC by TCP type.
    The only way that can help us is to remove the Source-NAT from this chain but how can we route traffic from external IP address to our internal IP addresses ?
  • diederikdiederik Member
    edited September 2018
    You do not need source nat to be able to route traffic from external IP addressing to internal IP adressing. Basic routing is done on destination address, and as long as the ADC knows how to forward the traffic to your internal network (I suppose it has an interface in your internal network) it will work.

    To forward traffic from the ADC to the servioce group, the destination address is altered. (Destination NAT). And that happens automatically when you have your ADC point the traffic to the IP addresses of the Envoy Proxy servers that are in your Service Group.

    Lets say we use a client from the internet which has a source in the 200.x.y.z range.
    And you service has a public IP address of 100.0.0.100

    The VIP on the ADC is 10.0.0.100 and the Envoy server the traffic gets forwarded to has 192.168.0.100 a private address.

    What you will see is that the external ip addressing is used for the outside session on the ADC and from the ADC without the Source-NAT it will only change the destination IP address.

    From client on the internet to the ADC
    external scr: 200.x.y.z
    external dst: 100.0.0.100

    From the ADC to the Envoy Proxy
    internal scr: 200.x.y.z
    internal dst: 192.168.0.100

    So when the traffic needs to go back from the proxy to the client om the internet, it will use the 200.x.y.z as destination address.
    Now you just need to make sure that that traffic passed from the Envoy Proxy through the A10 ADC back to the internet... when traffic returns over the A10 ADC, it will recognise the session and will Source-NAT it automatically back to the VIP address.

    From the Envoy Proxy to the ADC
    internal src: 192.168.0.100
    internal dst: 200.x.y.z

    From the ADC to the client on the internet
    external src: 100.0.0.100
    external dst: 200.x.y.z
  • flaneurflaneur Member
    edited September 2018
    Thank you for the great answer but I can't make it works.I have VIP with real external address and when disable Source-NAT to my internal network Virtual Service immediately refuses to work.In my routing table on the ADX side I have only one route from 0.0.0.0 /0 to my default gateway.Don't know what to try :(
  • diederikdiederik Member
    edited September 2018
    Ok, that is most likely because the Envoy proxies have a default gateway that is not the A10. Or maybe there is router between the A10 and the proxy?

    You need to take a look at the proxy and see where it sends the traffic that needs to go back to the internet. Check it's routing table.

    If the Proxy and the A10 share the same broadcast domain, then you need to set the default gateway of the proxy to the A10 internal address. I assume you have VRRP-A setup, so set it to the floating address of your A10 cluster.
    I assume this is the case as the A10 only has a default gateway set, and no specific routes for the internal network.

    If there is one or more routers between the A10 and the proxy, you also need to make sure that those routers send the traffic in the right direction.

    But, make sure that this does not break other traffic flows in your network.

    Understand that if you do this, all traffic towards the internet from the proxy will go towards the A10.
    So if there is a service on the proxy that needs to update things like signatures/URL Classification lists/or maybe needs to talk to other devices in your network that are not in a broadcast domain of one of the interfaces, you need to fix that as well. Maybe through specific routes or by enabling outbound NAT on the A10 for the proxies.

    I would strongly suggest you contact the A10 Business partner or your A10 Systems Engineer/Account Team so they can assist you with finding out what the best approach is here.
  • flaneurflaneur Member
    edited September 2018
    Unfortunately we don't have any support from A10 and this is pain.Our hosting company provide us AX1030 load balancer but it seems that they don't know much about it's configuration or they don't want to do this.
  • diederikdiederik Member
    edited September 2018
    Well, the routing part is not specific to A10.
    The issue you are having getting the traffic back from the proxy to the A10, would be the same if you had any other equipment in place of the A10.
    This is basic networking.

    All they have to do is make sure the traffic from the Proxy is send back to the A10.
    So the configuration of the proxy needs to change first... not the A10.

    But of course you can always contact A10 to see who the contacts are in your region.
    You could hire A10's Professional Services team to have a look.
  • ericnericn Member
    edited September 2018
    In a common setup:
    slb server PROXY1 <some_private_ip>
      stuff
    slb server PROXY2 <other_private_ip>
       stuff
    slb service-group MY_Proxy
       member PROXY1 <port/s>
       member PROXY2 <port/s>
    slb virtual-server MYVIP <some_public_ip>
       port <port/s> TCP
          service-group MY_PROXY

    Client connects to <some_public_ip>, and the ADC does the NAT (Destination NAT, rather than SourceNAT) to the private IPs inside/behind the ACOS device.

    So on the outside, you route to the public_IP, and on the inside, you set the ADC as the default gateway of the inside hosts.

    For most topologies in use today (L3 Routed w/o SNAT, L3 Routed w/ SNAT, and L2 One Arm Mode), ACOS is doing DNAT.
  • flaneurflaneur Member
    edited September 2018
    Yes,I have totally the same configuration as you described.But traffic didn't route from <some_public_ip> to some_private_network until I enable Soutce-NAT configuration.That's my problem now.I have also two interfaces in my ADC, one with Public IP and another one with Private IP.Can you suggest some ideas ?
  • diederikdiederik Member
    edited September 2018
    To understand what is going on, we need to have the interface configuration information from both the proxy and the A10 and the routing table on both.
  • flaneurflaneur Member
    edited September 2018
    Configuration on the server where Envoy proxy is installed:
    2 interfaces - public - ip network 212.32.x.x and private - ip network 10.201.0.x.
    routing table:
    Destination Gateway Genmask Flags Metric Ref Use Iface
    0.0.0.0 212.32.x.x 0.0.0.0 UG 0 0 0 eth4
    10.0.0.0 0.0.0.0 255.128.0.0 U 0 0 0 br-bdfaffd0acf2
    10.201.0.0 0.0.0.0 255.255.252.0 U 0 0 0 eth5
    172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
    212.32.252.64 0.0.0.0 255.255.255.192 U 0 0 0 eth4

    Configuration on the ADC side:
    2 interfaces - ethernet1 and ethernet2
    ethernet1 - 212.32.x.x/27 - public interface with static ip which I use to access ADC web ui
    ethernet2 - 10.201.0.250/22 private interface to communicate with private network.
    Routing map on ADC:
    0.0.0.0 0.0.0.0 212.32.252.158 ethernet 1 Static
    212.32.252.128 255.255.255.224 0.0.0.0 ethernet 1 Connected
    ADC was setuped in gateway mode.
    VRRP-A isn't used.
    What else information do you need ?
  • diederikdiederik Member
    edited September 2018
    If this route on the Envoy:
    0.0.0.0 212.32.x.x 0.0.0.0 UG 0 0 0 eth4

    Points towards the IP address of the A10 ADC then it should work.
    Unless something on your network is doing Proxy-ARP

    As the A10 does not have interfaces in the private IP range, you need to point the:
    slb server PROXY1 <some_private_ip>
      stuff
    slb server PROXY2 <other_private_ip>
       stuff

    Towards the public IP of the Envoy servers.

    Best is to do a packet trace on both the Envoy as well as on the A10 so you can exactly follow what is happening.
  • flaneurflaneur Member
    edited September 2018
    0.0.0.0 212.32.x.x 0.0.0.0 UG 0 0 0 eth4

    No this route isn't to ADC, it is to some device from our hoster,I'm now trying to get information from our hoster about what is it.So I assume this is our global issue.
  • flaneurflaneur Member
    edited September 2018
    Change the default gateway on the server with Envoy to A10 IP.Now I don't know how to setup A10 in transparent mode,can you help ?
  • diederikdiederik Member
    edited September 2018
    You do not need to set the A10 into transparent mode.
    I'm afraid this is not something that can easily be fixed using the forum.

    Somebody needs to have a look at the total setup suggest how all devices should be configured.

    I strongly suggest you to contact an A10 account team in your region so they can discuss the options to solve this.


    You can find local contact details here:
    https://www.a10networks.com/company/contact-us
  • flaneurflaneur Member
    edited September 2018
    Do you know if AX1030 ADC use HA Porxy PROXY protocol to send client IP trough TCP ?
    I mean option Insert Client IP in Config Mode > SLB > Template > TCP Proxy.
  • diederikdiederik Member
    edited September 2018
    Unfortunately, no, the A10 does not support the HA Proxy PROXY Protocol.

    It could possibly be build in aFlex, but then requires the port type HTTP.

    The option Client IP in the template TCP Proxy, uses the TCP Options.
    https://www.iana.org/assignments/tcp-parameters/tcp-parameters.xml
Sign In or Register to comment.