Application Network Visibility using the Prometheus and A10 Thunder
This article describes how to set up Thunder ADC and Prometheus to gain application network visibility and operational insights using a visualization tool like Grafana.
A10 Thunder supports a logging system to monitor resources like system (CPU, Memory usage), interface statistics, as well as service metrics and performance for application networking (ADC, CGN) and security services (FW, DDoS) through API. Starting from ACOS version 5.2, A10 Thunder supports Prometheus natively and data collection can be done directly without requiring a Prometheus Exporter.
Prometheus uses the HTTP PULL model to scrape such various data off of network devices and store the received values in a time-series database. Using a visualization and analytics tool, like Grafana, the data stored in Prometheus can be tracked to understand various baselines and thresholds that the networks should be operating under, thus enabling event monitoring and alerting.
In this example, we use Prometheus to directly collect following data from Thunder ADC, and Grafana to show dashboard for application network performance such as packets rate, throughput, connections for the service.
- VIP service port stats and rate
- Layer 7 (HTP-proxy) stats
- Backend server stats and rate
- System resource (CPU usage, memory usage, interface stats)
Prerequisites
- You will need previous experience with Prometheus, Grafana, and Thunder ADC
- Prometheus
- Install a Prometheus service on a server. In this example, Prometheus is running as a Docker image using port 9090 on a CentOS 7 virtual machine.
docker run -p 9090:9090 -v /{path_to_config}:/etc/prometheus prom/prometheus
- For Prometheus installation, refer to Prometheus Installation.
- Grafana
- Install a Grafana service on a server. In this example, Grafana is also running as a Docker image using port 3000 on the same server as the Prometheus.
docker run -d -p 3000:3000 grafana/grafana
- Add Prometheus as a data source
- For Grafana installation, refer to Grafana Installation.
- Thunder ADC
- Thunder ADC (hardware, virtual, bare metal or container) running ACOS version 5.2.1 or above.
- Get vThunder free trial
- Note: Thunder have native support for Prometheus in ACOS 5.2.0. A10 device running ACOS 3.x/4.x can be integrated with the use of Prometheus Exporter.
- Topology
- Thunder ADC is deployed to load balance multiple servers.
- A monitoring server is hosting Prometheus and Grafana services.
- Prometheus is pulling application traffic telemetry and statistics from Thunder ADC over management network.
Prometheus Configuration
The Prometheus configuration file prometheus.yml
defines everything related to scraping jobs and their instances, as well as which rule files to load. In this example, we define new job “a10-adc.101” under “scrape_configs” section to scrape Thunder ADC device for system and SLB service metrics as shown below.
- Stats and rate for VIP vip100
- Stats for service-group sg1
- Stats and rate for backend servers (s1 and s2)
- Stats for HTTP-Proxy service
- Interface stats
- CPU and memory usage
scrape_configs: - job_name: 'a10-adc.101' scheme: 'https' tls_config: insecure_skip_verify: true metrics_path: '/metrics' scrape_interval: 15s static_configs: - targets: ['10.64.4.101'] params: api_endpoint: ['/interface/stats','/system/data-cpu/stats','/system/control-cpu/stats','/system/memory/stats','/slb/virtual-server/vip100/stats','/slb/virtual-server/rate',’/slb/service-group/sg1/stats’,'/slb/server/stats','/slb/server/rate','/slb/http-proxy/stats'] username: ['prometheus'] password: ['prometheus']
Note: If no api_endoint is specified on the job for A10 Thunder, the following system metrics are collected by default.
- /interface/stats
- /system/control-cpu/stats
- /system/data-cpu/stats
- /system/memory/stats
Thunder ADC Configuration
There is no specific configuration required for Prometheus integration on the Thunder ADC.
Note: This example assumes the communication between Thunder and Prometheus are over management port. If you are using one of data interfaces to communicate with Prometheus, you need to allow “https” access using the “enable-management” command on the Thunder.
Since the goal of this is to obtain application networking visibility and insights, you may want to enable extended-stats
under slb common
and sampling-enable
on each slb objects (where needed) that enable collection of extended SLB stats including peak connection and rate information of each enabled counters over the intervals (1sec, 5sec, 10sec and 30sec) respectively.
Here is the sample ADC configuration with extended-stats
and sample-enable
configured. Please note that sampling-enable
is not configured on service group sg1 since the counter-rate
metrics for service group is not defined on Prometheus this time.
slb common extended-stats enable-l7-req-acct ! slb server s1 192.168.0.10 sampling-enable all port 80 tcp ! slb server s2 192.168.0.11 sampling-enable all port 80 tcp ! slb service-group sg1 tcp member s1 80 member s2 80 ! slb virtual-server vip100 10.10.10.100 port 80 http source-nat auto service-group sg1 sampling-enable all
As a tip, the most of API endpoint for SLB counters are equivalent to the output of the show counters(-rate) slb
command like shown below. You may find appropriate metrics and counters to be monitored with Prometheus and Grafana.
API endpoint: /slb/server/s1/stats
== show counters slb server s1
API endpoint: /slb/server/s1/rate
== show counters-rate slb server s1
Sample Prometheus Request:
https://10.64.4.101/metrics?api_endpoint=/slb/server/s1/stats&api_endpoint=/slb/server/s1/rate&password=prometheus&username=prometheus
ADC#show counters slb server s1 show counters slb server s1 ************************************** Current established connections 0 Total established connections 8850 Forward Packets Processed 61948 Reverse Packets Processed 44251 Peak number of established connections 1 Total Requests processed 8854 Total Requests succeeded 8854 Current SSL connections established 0 Total SSL connections established 0 Bytes processed in forward direction 4471071 Bytes processed in reverse direction 20212126 Packets processed in forward direction 61948 Packets processed in reverse direction 44251 IP-Only-LB Bytes processed in forward direction 0 IP-Only-LB Bytes processed in reverse direction 0 IP-Only-LB Packets processed in forward direction 0 IP-Only-LB Packets processed in reverse direction 0 ADC#show counters-rate slb server s1 counter_name 1sec_rate 5sec_rate 10sec_rate 30sec_rate Total established connections 0 4 4 4 Forward Packets Processed 0 32 32 31 Reverse Packets Processed 0 23 23 22 Peak number of established connections 0 0 0 0 Total Requests processed 5 4 4 4 Total Requests succeeded 5 4 4 4 Current SSL connections established 0 0 0 0 Total SSL connections established 0 0 0 0 Bytes processed in forward direction 2525 2323 2323 2272 Bytes processed in reverse direction 11415 10501 10501 10273 Packets processed in forward direction 0 32 32 31 Packets processed in reverse direction 0 23 23 22 IP-Only-LB Bytes processed in forward direction 0 0 0 0 IP-Only-LB Bytes processed in reverse direction 0 0 0 0 IP-Only-LB Packets processed in forward direction 0 0 0 0 IP-Only-LB Packets processed in reverse direction 0 0 0 0
Grafana Setup
Once Prometheus starts colleting metrics from Thunder, you can create your own dashboard to visualize application traffic and monitor device and service status using threshold and baselining.
In this example, following metrics are used for each data source.
- CPU
- cpu_n{data="system_control_cpu"}
- cpu_n{data="system_data_cpu"}
- Memory
- usage_percentage
- VIP user connection insights
- peak_conn
- VIP application traffic insights
- total_fwd_bytes
- total_rev_bytes
- throughput_bits_per_sec
- curr_conn
- curr_conn_rate
- Error traffic stats
- response_4xx, response_5xx, es_resp_invalid_http, es_total_failure_actions_rate
- Server traffic insights
- total_fwd_pkts_rate
- total_rev_pkts_rate
You want to watch a demo of the A10 Thunder and Prometheus integration? Go to a10transcend.com