Configuring xNIC on Endpoints for HA

  • Updated on Nov 13, 2024
  • Published on Sep 6, 2024
  • 15 minute(s) read
Prev Next

WHAT TO EXPECT

In this article, users will learn the various ways xNICs at their endpoints can be configured for high availability. While they can do it manually below, it is recommended to configure High Availability via the wXcked Eye Topology Graph.

Note: Reinstalling the xNIC will re-write configuration files to their original state, undoing any of the above configurations.

Configuring xNIC for High Availability

After a user sets up their cloudSwXtch for High Availability, the xNIC will automatically configure itself to receive and/or send HA traffic for a single multicast group. Users can confirm high availability has been configured by viewing the JSON file in the VM where their xNIC resides. 

Where to Find the Bridge JSON File

To configure the cloudSwXtch Bridge for high availability streams, the swxtch-bridge.json file needs to be updated. See where to locate them below:

For Linux:
The file can be found in /var/opt/swxtch/xnic.json. Currently, only V1 is supported for Linux.
To edit the file, one option is to use nano as shown below:

sudo nano /var/opt/swxtch/xnic.json

For Windows:
The file can be found in C:\Program Files\SwXtch.io\Swxtch-xNIC\xnic.json

Single Multicast Group

An example of the xnic.json file is shown below. Note the ha section has been added. 

{
    "swxtch": "10.2.128.10",
    "controlInterface": "Ethernet 2",
    "dataInterface": "Ethernet",
    "dataPort": 9999,
    "xnicType": 2,
    "dataPlaneSpecs": {
        "verbosity": 0,
        "virtualInterface": {
            "name": "swxtch-tun0",
            "ip": "172.30.0.0",
            "subnet": "255.255.0.0",
            "mtu": 4096
        },
        "bpfPrograms": [
        ]
    },
    "ha": {
        "enable": true,
        "protocol": "rtp",
        "deduplication": {
            "maxTimeToBufferPacketsMs": 50,
            "bufferSizeInPackets": 131072
        }
    },
      "overrideSrcIp": false,
      "statsReportWait": 60,
      "subscriptionsPollingIntervalMs": 100
}

ha Section Explained

The ha section exposes variables that can alter the behavior of the hitless switching code. The values for MaxTimeToBufferPackets_ms and BufferSizeInPackets in the example are good, suggested values; however, they can be tweaked to meet desired high availability requirements.

  • enable: If set to true, the xNIC will join the HA cluster when configured. The data and control traffic will start flowing to/from cloudSwXtches in the HA cluster. If set to false, the xNIC will ignore the HA cluster and continue operating only with its primary cloudSwXtch. The default is true as it is the most common use case.

  • Protocol- how to parse the packet. The available options are swxtch or rtp.

    • swxtch  = This can be used when the xNIC is duplicating or deduplicating the multicast. The xNIC will reconstruct based on the sequence count inside the cloudSwXtch packet header.

    • rtp = This should be used when processing RTP packets sent from a non-xNIC source. The xNIC will reconstruct based on the RTP timestamp information for Real-time Transport Protocol.

  • deduplication: This is only valid if “enable” is set to true. When this field is populated with the following values, the data plane will turn on HA and de-duplicate traffic from the HA cluster. If set to null or non-existent, it will receive all traffic from the HA cluster, allowing users to deduplicate using their own application.

    • MaxTimeToBufferPackets_ms - how long to buffer packets before declaring it as lost.

    • BufferSizeInPackets- the max number of packets that can be buffered.

  • subscriptionsPollingIntervalMs: Time in milliseconds to buffer and wait for reordered/delayed packets. Default: ~100ms

Below is an example of what a single multicast group configuration might look like. In this example, a user is sending the same multicast traffic (239.1.1.1:8804) via two paths with the xNIC consuming both and deduplicating at the end point. 

Multiple Multicast for xNIC

Important Rules

  • Protocol for Multiple Multicast (MMC) must be set to rtp.

  • For MMC on a producer, the xNIC must be Type 1.

  • If the producer is set to true, then it must be set to Type 1 even if you’re just consuming.

  • Multiple NICs are currently not supported for Ingress Windows Type 2.

If a user wants to set up for Multiple Multicast groups, they will need to manually configure the xnic.json file. Below is an example of what a multiple multicast group configuration might look like:

In this example, you have two paths with the same multicast traffic with different IP addresses. Path 1 is 239.1.1.1 while Path 2 is 239.1.1.2. The application at the end point is listening to 239.2.2.2, which is grouping together Path 1 and Path 2. The xNIC at the end point is tasked with deduplication.

A sample xnic.json file of the diagram is shown below with a "streamSpecs" section added.

Note that the following rules apply to this json file:

  • The xnicType is set to 2 but could also be 1. It has to be set to 1 if the VM is for a producer.

  • In the HA section, the protocol is set to rtp. 

{
    "swxtch": "10.2.128.10",
    "controlInterface": "Ethernet 2",
    "dataInterface": "Ethernet",
    "dataPort": 9999,
    "xnicType": 2,
    "dataPlaneSpecs": {
        "verbosity": 0,
        "virtualInterface": {
            "name": "swxtch-tun0",
            "ip": "172.30.0.0",
            "subnet": "255.255.0.0",
            "mtu": 4096
        },
        "bpfPrograms": [
        ]
    },
    "ha": {
        "enable": true,
        "protocol": "rtp",
        "deduplication": {
            "maxTimeToBufferPacketsMs": 50,
            "bufferSizeInPackets": 131072
        }
    },
    "streamSpecs": {
      "mmcProducerEnable": false,
      "multipleMulticastGroups": {
          "239.2.2.2:8804": {
              "pathStreams": [
                  {
                      "stream": "239.1.1.1:8804"
                  },
                  {
                      "stream": "239.1.1.2:8804"
                  }
              ]
          }
      }
},
  "overrideSrcIp": false,
  "statsReportWait": 60,
  "subscriptionsPollingIntervalMs": 100
}

Here, the user is grouping together 2 multicast IPs (239.1.1.1 and 239.1.1.2) and assigning it a multicast group IP address (239.2.2.2). The application at the endpoint is listening for 239.2.2.2, which the xNIC will deduplicate into a stream from 239.1.1.1 and 239.1.1.2. This was illustrated in the diagram above. 

Please note: At this time, the ports for the multicast group and the path streams must be the same. 

How to update xnic.json file for Multiple Multicast Groups

The user will have to make the following changes to their xnic.json file found in the single multicast group configuration to match the example above. These alterations to the xnic.json file should happen after Configuring the cloudSwXtch for High Availability. 

  1. The default xNIC Type is set to 2. Change the xNIC type to 1 if following conditions are true:  

    1. if the VM is a producer

    2. OR if the VM is a Windows machine with multiple data NICs. For more information about multiple data NICs, see here.

  2. Change the protocol under ha from "swxtch" to "rtp" including the quotation marks. 

  3. For each multicast group, add the following "streamSpecs" section as shown below with your stream data. If the VM is a producer, set mmcProducerEnable to true. Note: A user can enter multiple multicast groups. 

},
      "streamSpecs": {
      "mmcProducerEnable": false,
      "multipleMulticastGroups": {
          "239.2.2.2:8804": {
              "pathStreams": [
                  {
                      "stream": "239.1.1.1:8804"
                  },
                  {
                      "stream": "239.1.1.2:8804"
                  }
              ]
          }
      }
},

3. Save the xnic.json file. 

4. Restart the swXtch-NIC Control service. 

  • For Windows, go to the Task Manager and under the Services tab, select and restart swXtchNIC-Control. 

    swXtch-NIC%20Control%20Windows

  • For Linux, use the following command:

sudo systemctl restart swxtch-xnic-control

Source Specific Multiple Multicast

Important Rules

  • Protocol for Source Specific Multiple Multicast must be set to rtp.

  • For Source Specific Multiple Multicast on a producer, the xNIC must be Type 1.

  • If the producer is set to true, then it must be set to Type 1 even if you’re just consuming.

If a user wants to set up Source Specific Multiple Multicast groups, they will need to manually configure the xnic.json file. Below is an example of what a source specific multiple multicast group configuration might look like:

HA_MMC_SSM_diagram

In this example, you have two paths with the same multicast traffic with different multicast IP addresses along with their sources. Path 1 is 239.1.1.1 with a source of 172.1.1.1 while Path 2 is 239.1.1.2 with a source of 172.1.1.2. The application at the end point is listening to 239.2.2.2, which is grouping together Path 1 and Path 2. The xNIC at the end point is tasked with deduplication.

Producer

A sample producer xnic.json file of the diagram is shown below with a "streamSpecs" section added. Note that the following rules apply to this producer json file:

  • The xnicType is set to 1.

  • In the HA section, the protocol is set to rtp.

  • In the streamSpecs section, there there is no source specified since it is a producer. 

{
  "swxtch": "172.41.128.25",
  "controlInterface": "ens5",
  "dataInterface": "ens6",
  "dataPort": 9999,
  "xnicType": 1,
  "xnicRpcPort": 10002,
  "dataPlaneSpecs": {
    "verbosity": 0,
    "virtualInterface": {
      "type": "tun",
      "name": "swxtch-tun0",
      "ip": "172.30.0.0",
      "subnet": "255.255.0.0",
      "mtu": 4096
    },
    "bpfPrograms": [
      {
        "name": "tc-ingress",
        "interface": "ens6",
        "attachPoint": "BPF_TC_INGRESS"
      },
      {
        "name": "tc-egress",
        "interface": "ens6",
        "attachPoint": "BPF_TC_EGRESS"
      },
      {
        "name": "tc-forwarder",
        "interface": "ens5",
        "attachPoint": "BPF_TC_EGRESS"
      },
      {
        "name": "tc-forwarder",
        "interface": "ens7",
        "attachPoint": "BPF_TC_EGRESS"
      }
    ]
  },
    "ha": {
        "enable": true,
        "protocol": "rtp",
        "deduplication": {
            "maxTimeToBufferPacketsMs": 50,
            "bufferSizeInPackets": 131072
        }
  },
    "streamSpecs": {
      "mmcProducerEnable": true,
      "multipleMulticastGroups": {
          "224.2.2.2:8400": {
              "pathStreams": [
                  {
                      "stream": "224.2.2.3:8400"
                  },
                  {
                      "stream": "224.2.2.4:8400"
                  }
              ]
          }
      }
},
  "overrideSrcIp": false,
  "statsReportWait": 60,
  "subscriptionsPollingIntervalMs": 100
}

Consumer

A sample consumer xnic.json file of the diagram is shown below with a "streamSpecs" section added.

Note that the following rules apply to this consumer json file:

  • The xnicType is set to 2 but should be 1 if the VM is a Windows machine with multiple data NICs. For more information about multiple data NICs, see here.

  • In the HA section, the protocol is set to rtp.

  • In the streamSpecs section, there is a source specified since it is a consumer.

{
    "swxtch": "172.41.128.25",
    "controlInterface": "Ethernet 3",
    "dataInterface": "Ethernet 4",
    "dataPort": 9999,
    "xnicType": 2,
    "xnicRpcPort": 10002,
    "dataPlaneSpecs": {
        "verbosity": 0,
        "virtualInterface": {
            "type": "tun",
            "name": "swxtch-tun0",
            "ip": "172.30.0.0",
            "subnet": "255.255.0.0",
            "mtu": 4096
        }
    },
      "ha": {
        "enable": true,
        "protocol": "rtp",
        "deduplication": {
            "maxTimeToBufferPacketsMs": 50,
            "bufferSizeInPackets": 131072
        }
    },
  "streamSpecs": {
    "mmcProducerEnable": false,
    "multipleMulticastGroups": {
        "239.2.2.2:8400": {
            "overrideSourceIp": "172.1.1.1",
            "pathStreams": [
                {
                    "stream": "239.1.1.1:8400",
                    "ssm": {
                        "filter": "include",
                        "srcList": [
                            "172.1.1.1"
                        ]
                    }
                },
                {
                    "stream": "239.1.1.2:8400",
                    "ssm": {
                        "filter": "include",
                        "srcList": [
                            "172.1.1.2"
                        ]
                    }
                }
            ]
        }
      }
    },
    "overrideSrcIp": false,
    "statsReportWait": 60,
    "subscriptionsPollingIntervalMs":100
}

Here, you have two paths with the same multicast traffic with different multicast IP addresses along with their sources. Path 1 is 239.1.1.1 with a source of 172.1.1.1 while Path 2 is 239.1.1.2 with a source of 172.1.1.2. The application at the end point is listening to 239.2.2.2, which is grouping together Path 1 and Path 2. The xNIC at the end point is tasked with deduplication.

Please note: At this time, the ports for the multicast group and the path streams must be the same.

How to update xnic.json file for Source Specific Multiple Multicast Groups

The user will have to make the following changes to their xnic.json file found in the single multicast group configuration to match the example above. These alterations to the xnic.json file should happen after Configuring the cloudSwXtch for High Availability.

  1. Change the xNIC type to 1 if the VM is a producer OR if is a Windows machine with multiple data NICs. For more information about multiple data NICs, see here.

  2. Change the protocol under ha from "swxtch" to "rtp" including the quotation marks.

  3. Insert the following after the HA section. Then, for each additional multiple multicast group, add a new multiple multicast set. If the VM is a producer, set mmcProducerEnable to true. 

"streamSpecs": {
    "mmcProducerEnable": false,
    "multipleMulticastGroups": {
        "239.2.2.2:8400": {
            "overrideSourceIp": "172.1.1.1",
            "pathStreams": [
                {
                    "stream": "239.1.1.1:8400",
                    "ssm": {
                        "filter": "include",
                        "srcList": [
                            "172.1.1.1"
                        ]
                    }
                },
                {
                    "stream": "239.1.1.2:8400",
                    "ssm": {
                        "filter": "include",
                        "srcList": [
                            "172.1.1.2"
                        ]
                    }
                }
            ]
        }
      }
    },

3. Save the xnic.json file.

4. Restart the swXtch-NIC Control service.

  • For Windows, go to the Task Manager and under the Services tab, select and restart swXtchNIC-Control.

    swXtch-NIC%20Control%20Windows

  • For Linux, use the following command:

sudo systemctl restart swxtch-xnic-control

How to update xnic.json file to Disable Deduplication at the Consumer

Some users may want to disable deduplication by the xNIC in favor of their own application. To do this, navigate to the xnic.json file and add “null” after the deduplication flag, removing the bracketed maxTimeToBufferPacketsMs and bufferSizeInPackets, as seen in the example below: 

"ha": {
    "enable": true,
    "protocol": "rtp",
    "deduplication": null
  },

This will inform the xNIC at the consumer VM, receiving traffic from the HA cluster, to not deduplicate the streams. The user can now apply their own application to deduplicate the traffic if they so wish.  

Multiple VPC/Data NIC Support

swXtch.io supports multiple VPC/Data NICs for network isolation across two or more paths. In the example below, you have two data NICs for red and blue paths.

HA_MMC_MultipleDataNICs_diagram(1)

The NICs can belong to either the same VPC or entirely different ones. For better data isolation, it is recommended to have each path flow through different VPCs so that if one complete VPC goes down, traffic will still flow via the other. Another benefit is that since the NIC is part of path/region, no peering is required between VPCs.  

AWS Multi-VPC ENI Attachments Explained

For a detailed description of multi-VPC ENI attachments on AWS, see their documentation here: https://aws.amazon.com/about-aws/whats-new/2023/10/multi-vpc-eni-attachments/

Multiple VPC/Data NIC Rules

The following rules apply for Multiple VPC/Data NIC on xNIC:

  1. For Multiple VPC/Data NICs on a producer, the xNIC must be Type 1.

  2. If the xNIC is the producer, then single multicast group is supported.

  3. Windows consumer must be set to Type 1.

For rules regarding a specific high availability configuration, see the corresponding section: