Options

Application Network Visibility using the Prometheus and A10 Thunder

tmitsuhatatmitsuhata Member, Administrator admin
edited January 2022 in DevOps

This article describes how to set up Thunder ADC and Prometheus to gain application network visibility and operational insights using a visualization tool like Grafana.

A10 Thunder supports a logging system to monitor resources like system (CPU, Memory usage), interface statistics, as well as service metrics and performance for application networking (ADC, CGN) and security services (FW, DDoS) through API. Starting from ACOS version 5.2, A10 Thunder supports Prometheus natively and data collection can be done directly without requiring a Prometheus Exporter.  

Prometheus uses the HTTP PULL model to scrape such various data off of network devices and store the received values in a time-series database. Using a visualization and analytics tool, like Grafana, the data stored in Prometheus can be tracked to understand various baselines and thresholds that the networks should be operating under, thus enabling event monitoring and alerting.

In this example, we use Prometheus to directly collect following data from Thunder ADC, and Grafana to show dashboard for application network performance such as packets rate, throughput, connections for the service.

  • VIP service port stats and rate
  • Layer 7 (HTP-proxy) stats
  • Backend server stats and rate  
  • System resource (CPU usage, memory usage, interface stats)

Prerequisites

  • You will need previous experience with Prometheus, Grafana, and Thunder ADC
  • Prometheus
    • Install a Prometheus service on a server. In this example, Prometheus is running as a Docker image using port 9090 on a CentOS 7 virtual machine.
    • docker run -p 9090:9090 -v /{path_to_config}:/etc/prometheus prom/prometheus
    • For Prometheus installation, refer to Prometheus Installation.
  • Grafana
    • Install a Grafana service on a server. In this example, Grafana is also running as a Docker image using port 3000 on the same server as the Prometheus.
    • docker run -d -p 3000:3000 grafana/grafana
    • Add Prometheus as a data source
    • For Grafana installation, refer to Grafana Installation.
  • Thunder ADC
    • Thunder ADC (hardware, virtual, bare metal or container) running ACOS version 5.2.1 or above.
    • Get vThunder free trial
    • Note: Thunder have native support for Prometheus in ACOS 5.2.0. A10 device running ACOS 3.x/4.x can be integrated with the use of Prometheus Exporter.
  • Topology
    • Thunder ADC is deployed to load balance multiple servers.
    • A monitoring server is hosting Prometheus and Grafana services.
    • Prometheus is pulling application traffic telemetry and statistics from Thunder ADC over management network.
a10_prometheus_native_integration.png

Prometheus Configuration

The Prometheus configuration file prometheus.yml defines everything related to scraping jobs and their instances, as well as which rule files to load. In this example, we define new job “a10-adc.101” under “scrape_configs” section to scrape Thunder ADC device for system and SLB service metrics as shown below.

  • Stats and rate for VIP vip100
  • Stats for service-group sg1
  • Stats and rate for backend servers (s1 and s2)
  • Stats for HTTP-Proxy service
  • Interface stats
  • CPU and memory usage  
scrape_configs:
 - job_name: 'a10-adc.101'
   scheme: 'https'
   tls_config:
     insecure_skip_verify: true
   metrics_path: '/metrics'
   scrape_interval: 15s
   static_configs:
   - targets: ['10.64.4.101']
   params:
     api_endpoint: ['/interface/stats','/system/data-cpu/stats','/system/control-cpu/stats','/system/memory/stats','/slb/virtual-server/vip100/stats','/slb/virtual-server/rate',’/slb/service-group/sg1/stats’,'/slb/server/stats','/slb/server/rate','/slb/http-proxy/stats']
     username: ['prometheus']
     password: ['prometheus']

Note: If no api_endoint is specified on the job for A10 Thunder, the following system metrics are collected by default.

  • /interface/stats
  • /system/control-cpu/stats
  • /system/data-cpu/stats
  • /system/memory/stats
a10_prometheus.png

Thunder ADC Configuration

There is no specific configuration required for Prometheus integration on the Thunder ADC.

Note: This example assumes the communication between Thunder and Prometheus are over management port. If you are using one of data interfaces to communicate with Prometheus, you need to allow “https” access using the “enable-management” command on the Thunder.

Since the goal of this is to obtain application networking visibility and insights, you may want to enable extended-stats under slb common and sampling-enable on each slb objects (where needed) that enable collection of extended SLB stats including peak connection and rate information of each enabled counters over the intervals (1sec, 5sec, 10sec and 30sec) respectively.  

Here is the sample ADC configuration with extended-stats and sample-enable configured. Please note that sampling-enable is not configured on service group sg1 since the counter-rate metrics for service group is not defined on Prometheus this time.   

slb common
 extended-stats
 enable-l7-req-acct
!               
slb server s1 192.168.0.10
 sampling-enable all
 port 80 tcp
!
slb server s2 192.168.0.11
 sampling-enable all
 port 80 tcp
!
slb service-group sg1 tcp
 member s1 80
 member s2 80
!
slb virtual-server vip100 10.10.10.100
 port 80 http
   source-nat auto
   service-group sg1
   sampling-enable all

As a tip, the most of API endpoint for SLB counters are equivalent to the output of the show counters(-rate) slb command like shown below. You may find appropriate metrics and counters to be monitored with Prometheus and Grafana.

API endpoint: /slb/server/s1/stats == show counters slb server s1

API endpoint: /slb/server/s1/rate == show counters-rate slb server s1

Sample Prometheus Request: 

https://10.64.4.101/metrics?api_endpoint=/slb/server/s1/stats&api_endpoint=/slb/server/s1/rate&password=prometheus&username=prometheus



ADC#show counters slb server s1
 
show counters slb server s1
**************************************
Current established connections                                 0
Total established connections                                   8850
Forward Packets Processed                                       61948
Reverse Packets Processed                                       44251
Peak number of established connections                          1
Total Requests processed                                        8854
Total Requests succeeded                                        8854
Current SSL connections established                             0
Total SSL connections established                               0
Bytes processed in forward direction                            4471071
Bytes processed in reverse direction                            20212126
Packets processed in forward direction                          61948
Packets processed in reverse direction                          44251
IP-Only-LB Bytes processed in forward direction                 0
IP-Only-LB Bytes processed in reverse direction                 0
IP-Only-LB Packets processed in forward direction               0
IP-Only-LB Packets processed in reverse direction               0
 
 
ADC#show counters-rate slb server s1
 
counter_name                                     1sec_rate          5sec_rate          10sec_rate         30sec_rate
Total established connections                    0                  4                  4                  4
Forward Packets Processed                        0                  32                 32                 31
Reverse Packets Processed                        0                  23                 23                 22
Peak number of established connections           0                  0                  0                  0
Total Requests processed                         5                  4                  4                  4
Total Requests succeeded                         5                  4                  4                  4
Current SSL connections established              0                  0                  0                  0
Total SSL connections established                0                  0                  0                  0
Bytes processed in forward direction             2525               2323               2323               2272
Bytes processed in reverse direction             11415              10501              10501              10273
Packets processed in forward direction           0                  32                 32                 31
Packets processed in reverse direction           0                  23                 23                 22
IP-Only-LB Bytes processed in forward direction  0                  0                  0                  0
IP-Only-LB Bytes processed in reverse direction  0                  0                  0                  0
IP-Only-LB Packets processed in forward direction 0                  0                  0                  0
IP-Only-LB Packets processed in reverse direction 0                  0                  0                  0

Grafana Setup

Once Prometheus starts colleting metrics from Thunder, you can create your own dashboard to visualize application traffic and monitor device and service status using threshold and baselining.

 In this example, following metrics are used for each data source.  

  • CPU
    • cpu_n{data="system_control_cpu"} 
    • cpu_n{data="system_data_cpu"}  
  • Memory
    • usage_percentage
  • VIP user connection insights 
    • peak_conn
  • VIP application traffic insights 
    • total_fwd_bytes 
    • total_rev_bytes 
    • throughput_bits_per_sec
    • curr_conn
    • curr_conn_rate
  • Error traffic stats
    • response_4xx, response_5xx, es_resp_invalid_http, es_total_failure_actions_rate
  • Server traffic insights
    • total_fwd_pkts_rate
    • total_rev_pkts_rate
a10_grafana.png


You want to watch a demo of the A10 Thunder and Prometheus integration? Go to a10transcend.com

Tagged:
Sign In or Register to comment.