Prometheus Monitoring

WHAT TO EXPECT

In this article, users will learn how to integrate Prometheus and Grafana as an additional way to monitor their cloudSwXtch environment.

PREREQUISITES

The following process assumes that you already have Prometheus and Grafana installed in a docker container.

STEP ONE: Validate cloudSwXtch can create Prometheus data

On the cloudSwXtch VM, run the following command:

curl http://localhost/prometheus/metrics

The output will list information about each metric with example output data. Metrics starting with swx_core are from the cloudSwXtch while metrics starting with swx_xnic are from xNICs. Since this example has only one cloudSwXtch but many VMs with xNICs, the xNIC data has multiple sample rows. Note that for brevity some of the xNIC rows returned have been deleted.

swxtchadmin@dsd-core-100:~$ curl http://localhost/prometheus/metrics
# HELP swx_core_droppedPacketCountByByteLimit Bytes dropped in the swXtch
# TYPE swx_core_droppedPacketCountByByteLimit counter
swx_core_droppedPacketCountByByteLimit{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_droppedPacketCountByPacketLimit Packets dropped in the swXtch
# TYPE swx_core_droppedPacketCountByPacketLimit counter
swx_core_droppedPacketCountByPacketLimit{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_rxBridgeByteCount Bridge bytes received into the swXtch
# TYPE swx_core_rxBridgeByteCount counter
swx_core_rxBridgeByteCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_rxBridgePacketCount Bridge packets received into the swXtch
# TYPE swx_core_rxBridgePacketCount counter
swx_core_rxBridgePacketCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_rxByteCount Bytes received into the swXtch
# TYPE swx_core_rxByteCount counter
swx_core_rxByteCount{category="swxtch_repl",host="10.1.216.23"} 1.10391744e+08
# HELP swx_core_rxMeshByteCount Mesh bytes received into the swXtch
# TYPE swx_core_rxMeshByteCount counter
swx_core_rxMeshByteCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_rxMeshPacketCount Mesh packets received into the swXtch
# TYPE swx_core_rxMeshPacketCount counter
swx_core_rxMeshPacketCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_rxPacketCount Packets received into the swXtch
# TYPE swx_core_rxPacketCount counter
swx_core_rxPacketCount{category="swxtch_repl",host="10.1.216.23"} 574957
# HELP swx_core_rxUnicastByteCount Unicast bytes received into the swXtch
# TYPE swx_core_rxUnicastByteCount counter
swx_core_rxUnicastByteCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_rxUnicastPacketCount Unicast packets received into the swXtch
# TYPE swx_core_rxUnicastPacketCount counter
swx_core_rxUnicastPacketCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_sequence swXtch sequence number
# TYPE swx_core_sequence counter
swx_core_sequence{category="swxtch_repl",host="10.1.216.23"} 3022
# HELP swx_core_txBridgeByteCount Bridge bytes sent from the swXtch
# TYPE swx_core_txBridgeByteCount counter
swx_core_txBridgeByteCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_txBridgePacketCount Bridge packets sent from the swXtch
# TYPE swx_core_txBridgePacketCount counter
swx_core_txBridgePacketCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_txByteCount Bytes sent from the swXtch
# TYPE swx_core_txByteCount counter
swx_core_txByteCount{category="swxtch_repl",host="10.1.216.23"} 1.09192128e+08
# HELP swx_core_txMeshByteCount Mesh bytes sent from the swXtch
# TYPE swx_core_txMeshByteCount counter
swx_core_txMeshByteCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_txMeshPacketCount Mesh packets sent from the swXtch
# TYPE swx_core_txMeshPacketCount counter
swx_core_txMeshPacketCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_txPacketCount Packets sent from the swXtch
# TYPE swx_core_txPacketCount counter
swx_core_txPacketCount{category="swxtch_repl",host="10.1.216.23"} 568709
# HELP swx_core_txUnicastByteCount Unicast bytes sent from the swXtch
# TYPE swx_core_txUnicastByteCount counter
swx_core_txUnicastByteCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_core_txUnicastPacketCount Unicast packets sent from the swXtch
# TYPE swx_core_txUnicastPacketCount counter
swx_core_txUnicastPacketCount{category="swxtch_repl",host="10.1.216.23"} 0
# HELP swx_maxClientCount Maximum number of clients by license
# TYPE swx_maxClientCount gauge
swx_maxClientCount{category="swxtch"} 10
# HELP swx_numClientsConnected Number of client currently connected
# TYPE swx_numClientsConnected gauge
swx_numClientsConnected{category="swxtch"} 2
# HELP swx_xnic_byteCounters_rxTotalCount Total bytes received from the swXtch into the component
# TYPE swx_xnic_byteCounters_rxTotalCount counter
swx_xnic_byteCounters_rxTotalCount{category="swxtch_xnic",host="10.1.213.120"} 0
swx_xnic_byteCounters_rxTotalCount{category="swxtch_xnic",host="10.1.213.121"} 914880
# HELP swx_xnic_byteCounters_txTotalCount Total bytes sent from the component into the swXtch
# TYPE swx_xnic_byteCounters_txTotalCount counter
swx_xnic_byteCounters_txTotalCount{category="swxtch_xnic",host="10.1.213.120"} 1.080768e+06
swx_xnic_byteCounters_txTotalCount{category="swxtch_xnic",host="10.1.213.121"} 0
# HELP swx_xnic_packetCounters_rxDroppedCount Lost packets received from the swXtch into the component
# TYPE swx_xnic_packetCounters_rxDroppedCount counter
swx_xnic_packetCounters_rxDroppedCount{category="swxtch_xnic",host="10.1.213.120"} 0
swx_xnic_packetCounters_rxDroppedCount{category="swxtch_xnic",host="10.1.213.121"} 0
# HELP swx_xnic_packetCounters_rxTotalCount Total packets received from the swXtch into the component
# TYPE swx_xnic_packetCounters_rxTotalCount counter
swx_xnic_packetCounters_rxTotalCount{category="swxtch_xnic",host="10.1.213.120"} 0
swx_xnic_packetCounters_rxTotalCount{category="swxtch_xnic",host="10.1.213.121"} 4765
# HELP swx_xnic_packetCounters_txDroppedCount Lost packets sent from the component into the swXtch
# TYPE swx_xnic_packetCounters_txDroppedCount counter
swx_xnic_packetCounters_txDroppedCount{category="swxtch_xnic",host="10.1.213.120"} 9606
swx_xnic_packetCounters_txDroppedCount{category="swxtch_xnic",host="10.1.213.121"} 0
# HELP swx_xnic_packetCounters_txTotalCount Total packets sent from the component into the swXtch
# TYPE swx_xnic_packetCounters_txTotalCount counter
swx_xnic_packetCounters_txTotalCount{category="swxtch_xnic",host="10.1.213.120"} 5629
swx_xnic_packetCounters_txTotalCount{category="swxtch_xnic",host="10.1.213.121"} 0
# HELP swx_xnic_rxMulticastGroups_byteCount Multicast group traffic sent from the Swxtch into the xNIC - Multicast group byte count
# TYPE swx_xnic_rxMulticastGroups_byteCount counter
swx_xnic_rxMulticastGroups_byteCount{category="swxtch_xnic",cloudSwxtchVersion="dev.f91a2e",groupIp="239.4.0.1",host="10.1.213.121",hostAddress="10.1.213.121",index="0",osDistribution="Ubuntu 20.04",swx_xnic_rxMulticastGroups_index="0",xNicType="t2"} 914880
# HELP swx_xnic_rxMulticastGroups_packetCount Multicast group traffic sent from the Swxtch into the xNIC - Multicast group packet count
# TYPE swx_xnic_rxMulticastGroups_packetCount counter
swx_xnic_rxMulticastGroups_packetCount{category="swxtch_xnic",cloudSwxtchVersion="dev.f91a2e",groupIp="239.4.0.1",host="10.1.213.121",hostAddress="10.1.213.121",index="0",osDistribution="Ubuntu 20.04",swx_xnic_rxMulticastGroups_index="0",xNicType="t2"} 4765
# HELP swx_xnic_txMulticastGroups_byteCount Multicast group traffic sent from the xNIC into the Swxtch - Multicast group byte count
# TYPE swx_xnic_txMulticastGroups_byteCount counter
swx_xnic_txMulticastGroups_byteCount{category="swxtch_xnic",cloudSwxtchVersion="dev.f91a2e",groupIp="239.4.0.1",host="10.1.213.120",hostAddress="10.1.213.120",index="0",osDistribution="Ubuntu 20.04",swx_xnic_txMulticastGroups_index="0",xNicType="t2"} 1.080768e+06
# HELP swx_xnic_txMulticastGroups_packetCount Multicast group traffic sent from the xNIC into the Swxtch - Multicast group packet count
# TYPE swx_xnic_txMulticastGroups_packetCount counter
swx_xnic_txMulticastGroups_packetCount{category="swxtch_xnic",cloudSwxtchVersion="dev.f91a2e",groupIp="239.4.0.1",host="10.1.213.120",hostAddress="10.1.213.120",index="0",osDistribution="Ubuntu 20.04",swx_xnic_txMulticastGroups_index="0",xNicType="t2"} 5629

If successful (there is an output), continue on to updating your Prometheus Directory for cloudSwXtch.

STEP TWO: Update Prometheus Directory for cloudSwXtch

Attached is an example prometheus.yaml file with an cloudSwXtch job name configuration.

1. Open the example prometheus.yml file and copy lines 26-36.

2. Paste those lines into your existing prometheus.yml file.

3. Update the cloudSwXtch targets line that has “ 127.0.0.1:80” and put in the IP address of the cloudSwXtch in place of the localhost IP.

   1. Please note: If Prometheus and cloudSwXtch are on the same VM, then the localhost IP (127.0.0.1) will still work.

      

4. Run the following docker command to run it in VM. If you decide to run it this way, you will need to run it after every reboot or when you close your window. Please use this method when testing in order to limit the amount of records added to the Prometheus database.

   docker run --network host -v ~/prometheus/:/etc/prometheus prom/prometheus

5. Use this command to run Prometheus automatically upon reboot. (Preferred method for a production environment.)

   docker run --network host --restart always -v ~/prometheus/:/etc/prometheus prom/prometheus

STEP THREE: Access Prometheus UI

In order to access the Prometheus UI, users should open a browser on their Windows machine in the same VNET and enter the following URL:

http://<prometheus-IP>:9090/

Note: Replace the <prometheus-IP> with the IP address of your Prometheus instance.

Users can enter the prefix “swx” into the search field to see a list of data fields related to the cloudSwXtch (swx_core) and its xNICs (swx_xnic).

In the example below, the user has chosen swx_core_rxByteCount as their data field.

By selecting Execute, users will be able to populate the table with the desired data. Note: Producers and consumers must be running in order to see data. In the example above, a user can select Execute multiple times and notice that the number in the orange box grow in size.

For a list of available data fields, view the dropdown section here:

http://<core>/prometheus/metrics

STEP FOUR: Access Grafana UI

Alternatively, users can also use Grafana as another method for viewing cloudSwXtch metrics.

1. In a Windows machine on the same VNET, open a browser and enter the following URL:

http://<Grafana-IP>:3000/

Note: Replace the <Grafana-IP> with the IP address of your Grafana instance.

2. Sign in as a user admin and the password admin.

3. Click on the three horizontal lines next to Home to get additional options.

4. Select Connection.

5. Click Add new connection.

   

6. Search Prometheus on the Add new connection page and select it from the options.

   

7. Click Create a Prometheus data source.

   

8. Go to Data Sources and select the Prometheus data source created.

9. Enter the Prometheus server URL. This should include the IP address of your Prometheus instance.

   

10. Click Explore.

    

11. In the search bar under Metric, use the prefix “swx” to populate a list of potential data fields.

    

12. Scroll through the list and select what is desired.

13. Run the query. The results will appear below.

    

For xNIC-related metrics, users can specify a host under Label Filters and select an agent. After selecting an agent, a user will need to run the query again to populate the graph as shown above.

If desired, a user can also change the amount of time in the top of the window.