BMWG L. Huang, Ed. Internet-Draft R. Gu, Ed. Intended status: Informational China Mobile Expires: January 18, 2018 Bob. Mandeville Iometrix Brooks. Hickman Spirent Communications July 17, 2017 Benchmarking Methodology for Virtualization Network Performance draft-huang-bmwg-virtual-network-performance-03 Abstract As the virtual network has been widely established in IDC, the performance of virtual network has become a valuable consideration to the IDC managers. This draft introduces a benchmarking methodology for virtualization network performance based on virtual switch. Status of This Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 18, 2018. Copyright Notice Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Huang, et al. Expires January 18, 2018 [Page 1] Internet-Draft virtual-network-performance-benchmark-03 July 2017 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Test Considerations . . . . . . . . . . . . . . . . . . . . . 3 4. Key Performance Indicators . . . . . . . . . . . . . . . . . 5 5. Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . 6 6. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 7 6.1. Throughput . . . . . . . . . . . . . . . . . . . . . . . 7 6.1.1. Objectives . . . . . . . . . . . . . . . . . . . . . 7 6.1.2. Configuration parameters . . . . . . . . . . . . . . 7 6.1.3. Test parameters . . . . . . . . . . . . . . . . . . . 8 6.1.4. Test process . . . . . . . . . . . . . . . . . . . . 8 6.1.5. Test result format . . . . . . . . . . . . . . . . . 8 6.2. Frame loss rate . . . . . . . . . . . . . . . . . . . . . 8 6.2.1. Objectives . . . . . . . . . . . . . . . . . . . . . 9 6.2.2. Configuration parameters . . . . . . . . . . . . . . 9 6.2.3. Test parameters . . . . . . . . . . . . . . . . . . . 9 6.2.4. Test process . . . . . . . . . . . . . . . . . . . . 9 6.2.5. Test result format . . . . . . . . . . . . . . . . . 10 6.3. CPU consumption . . . . . . . . . . . . . . . . . . . . . 10 6.3.1. Objectives . . . . . . . . . . . . . . . . . . . . . 10 6.3.2. Configuration parameters . . . . . . . . . . . . . . 10 6.3.3. Test parameters . . . . . . . . . . . . . . . . . . . 11 6.3.4. Test process . . . . . . . . . . . . . . . . . . . . 11 6.3.5. Test result format . . . . . . . . . . . . . . . . . 11 6.4. MEM consumption . . . . . . . . . . . . . . . . . . . . . 12 6.4.1. Objectives . . . . . . . . . . . . . . . . . . . . . 12 6.4.2. Configuration parameters . . . . . . . . . . . . . . 12 6.4.3. Test parameters . . . . . . . . . . . . . . . . . . . 13 6.4.4. Test process . . . . . . . . . . . . . . . . . . . . 13 6.4.5. Test result format . . . . . . . . . . . . . . . . . 13 6.5. Latency . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.5.1. Objectives . . . . . . . . . . . . . . . . . . . . . 15 6.5.2. Configuration parameters . . . . . . . . . . . . . . 15 6.5.3. Test parameters . . . . . . . . . . . . . . . . . . . 15 6.5.4. Test process . . . . . . . . . . . . . . . . . . . . 15 6.5.5. Test result format . . . . . . . . . . . . . . . . . 15 7. Security Considerations . . . . . . . . . . . . . . . . . . . 16 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 9. Normative References . . . . . . . . . . . . . . . . . . . . 16 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 1. Introduction As the virtual network has been widely established in IDC, the performance of virtual network has become a valuable consideration to the IDC managers. This draft introduces a benchmarking methodology Huang, et al. Expires January 18, 2018 [Page 2] Internet-Draft virtual-network-performance-benchmark-03 July 2017 for virtualization network performance based on virtual switch as the DUT. 2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 3. Test Considerations In a conventional test setup with Non-Virtual test ports, it is quite legitimate to assume that test ports provide the golden standard in measuring the performance metrics. If test results are sub optimal, it is automatically assumed that the Device-Under-Test (DUT) is at fault. For example, when testing throughput at a given frame size, if the test result shows less than 100% throughput, we can safely conclude that it's the DUT that can not deliver line rate forwarding at that frame size(s). We never doubt that the tester can be an issue. While in a virtual test environment where both the DUT as well as the test tool itself are software based, it's quite a different story. Just like the DUT, tester running as software will have its own performance peak under various conditions. There are two types of vSwitch according to different installation location. One is VM based vSwitch which is installed on a virtual machine, another is vSwitch directly installed on the host OS (similar to hypervisor).The latter is much more popular currently. Tester's calibration is essential in benchmarking testing in a virtual environment. Furthermore, to reduce the enormous combination of various conditions, tester must be calibrated with the exact same combination and parameter settings the user wants to measure against the DUT. A slight variation of conditions and parameter values will cause inaccurate measurements of the DUT. While it is difficult to list the exact combination and parameter settings, the following table attempts to give the most common example how to calibrate a tester before testing a DUT (VSWITCH). Sample calibration permutation: Huang, et al. Expires January 18, 2018 [Page 3] Internet-Draft virtual-network-performance-benchmark-03 July 2017 ---------------------------------------------------------------- | Hypervisor | VM VNIC | VM Memory | Frame | | | Type | Speed |CPU Allocation | Size | Throughput| ---------------------------------------------------------------- | ESXi 1G/10G 512M/1Core | 64 | | | | 128 | | | | 256 | | | | 512 | | | | 1024 | | | | 1518 | | ---------------------------------------------------------------- Figure 1: Sample Calibration Permutation Key points are as following: a) The hypervisor type is of ultimate importance to the test results. VM tester(s) MUST be installed on the same hypervisor type as the DUT (VSWITCH). Different hypervisor type has an influence on the test result. b) The VNIC speed will have an impact on testing results. Testers MUST calibrate against all VNIC speeds. c) VM allocations of CPU resources and memory have an influence on test results. d) Frame sizes will affect the test results dramatically due to the nature of virtual machines. e) Other possible extensions of above table: The number of VMs to be created, latency reading, one VNIC per VM vs. multiple VM sharing one VNIC, and uni-directional traffic vs. bi-directional traffic. Besides, the compute environment including the hardware should be also recorded. Huang, et al. Expires January 18, 2018 [Page 4] Internet-Draft virtual-network-performance-benchmark-03 July 2017 ----------------------------------------------------- | Compute encironment componenets | Model | ----------------------------------------------------- | CPU | | ----------------------------------------------------- | Memory | | ----------------------------------------------------- | Hard Disk | | ----------------------------------------------------- | 10G Adaptors | | ----------------------------------------------------- | Blade/Motherboard | | ----------------------------------------------------- Figure 2: Compute Environment It's important to confirm test environment for tester's calibration as close to the environment a virtual DUT (VSWITCH) involved in for the benchmark test. Key points which SHOULD be noticed in test setup are listed as follows. 1. One or more VM tester(s) need to be created for both traffic generation and analysis. 2. vSwitch has an influence on performance penalty due to extra resource occupation. 3. VNIC and its type is needed in the test setup to once again accommodate performance penalty when DUT (VSWITCH) is created. In summary, calibration should be done in such an environment that all possible factors which may negatively impact test results should be taken into consideration. 4. Key Performance Indicators We listed numbers of key performance indicators for virtual network below: a) Throughput under various frame sizes: forwarding performance under various frame sizes is a key performance indicator of interest. b) DUT consumption of CPU: when adding one or more VM(s), DUT (VSWITCH) will consume more CPU. Vendors can allocate appropriate CPU to reach the line rate performance. Huang, et al. Expires January 18, 2018 [Page 5] Internet-Draft virtual-network-performance-benchmark-03 July 2017 c) DUT consumption of MEM: when adding one or more VM(s), DUT (VSWITCH) will consume more memory. Vendors can allocate appropriate MEM to reach the line rate performance. d) Latency readings: Some applications are highly sensitive on latency. It's important to get the latency reading with respective to various conditions. Other indicators such as VxLAN maximum supported by the virtual switch and so on can be added in the scene when VxLAN is needed. 5. Test Setup The test setup is classified into two traffic models: Model A and Model B. In traffic model A: A physical tester connects to the server which bears the DUT (VSWITCH) and Virtual tester to verify the benchmark of server. ________________________________________ | | ----------------- | ---------------- ---------------- | |Physical tester|------|---|DUT (VSWITCH) |----|Virtual tester| | ----------------- | ---------------- ---------------- | | Server | |________________________________________| Figure 3: test model A In traffic model B: Two virtual testers are used to verify the benchmark. In this model, two testers are installed in one server. ______________________________________________________________ | | | ---------------- ---------------- ---------------- | | |Virtual tester|----|DUT (VSWITCH) |-----|Virtual tester| | | ---------------- ---------------- ---------------- | | Server | |______________________________________________________________| Figure 4: test model B In our test, the test bed is constituted by physical servers of the Dell with a pair of 10GE NIC and physical tester. Virtual tester which occupies 2 vCPU and 8G MEM and DUT (VSWITCH) are installed in the server. 10GE switch and 1GE switch are used for test traffic and management respectively. Huang, et al. Expires January 18, 2018 [Page 6] Internet-Draft virtual-network-performance-benchmark-03 July 2017 This test setup is also available in the VxLAN measurement. 6. Benchmarking Tests 6.1. Throughput Unlike traditional test cases where the DUT and the tester are separated, virtual network test has been brought in unparalleled challenges. In virtual network test, the virtual tester and the DUT (VSWITCH) are in one server which means they are physically converged, so the test and DUT (VSWITCH) are sharing the same CPU and MEM resources of one server. Theoretically, the virtual tester's operation may have influence on the DUT (VSWITCH)'s performance. However, for the specialty of virtualization, this method is the only way to test the performance of a virtual DUT. Under the background of existing technology, when we test the virtual switch's throughput, the concept of traditional physical switch CANNOT be applicable. The traditional throughput indicates the switches' largest forwarding capability, for certain bytes selected and under zero-packet-lose conditions. But in virtual environments, virtual variations on virtual network will be much greater than that of dedicated physical devices. As the DUT and the tester cannot be separated, it proves that the DUT (VSWITCH) realize such network performances under certain circumstances. Therefore, we change the bytes in virtual environment to test the maximum value which we think of the indicator of throughput. It's conceivable that the throughput should be tested on both the test model A and B. The tested throughput has certain referential meanings to value the performance of the virtual DUT. 6.1.1. Objectives The objective of the test is to determine the throughput of the DUT (VSWITCH), which the DUT can support. 6.1.2. Configuration parameters Network parameters should be defined as follows: a) the number of virtual tester (VMs) b) the number of vNIC of virtual tester c) the CPU type of the server d) vCPU allocated for virtual tester (VMs) Huang, et al. Expires January 18, 2018 [Page 7] Internet-Draft virtual-network-performance-benchmark-03 July 2017 e) memory allocated for virtual tester (VMs) f) the number and rate of server NIC 6.1.3. Test parameters a) test repeated times b) test frame length 6.1.4. Test process 1. Configure the VM tester to offer traffic to the V-Switch. 2. Increase the traffic rate of tester until packet loss happens. 3. Record the max traffic rate without packet loss on VSwitch. 4. Change the frame length and repeat from step1 to step4. 6.1.5. Test result format -------------------------- | Byte| Throughput (Gbps)| -------------------------- | 64 | | -------------------------- | 128 | | -------------------------- | 256 | | -------------------------- | 512 | | -------------------------- | 1024| | -------------------------- | 1518| | -------------------------- Figure 5: test result format 6.2. Frame loss rate Frame loss rate is also an important indicator in evaluating the performance of virtual switch.As is defined in RFC 1242, percentage of frames that should have been forwarded which actually fails to be forwarded due to lack of resources needs to be tested.Both model A Huang, et al. Expires January 18, 2018 [Page 8] Internet-Draft virtual-network-performance-benchmark-03 July 2017 and model B are tested.Frame loss rate is an important indicator in evaluating the performance of virtual switches. 6.2.1. Objectives The objective of the test is to determine the frame loss rate under different data rates and frame sizes.. 6.2.2. Configuration parameters Network parameters should be defined as follows: a) the number of virtual tester (VMs) b) the number of vNIC of virtual tester c) the CPU type of the server d) vCPU allocated for virtual tester (VMs) e) memory allocated for virtual tester (VMs) f) the number and rate of server NIC 6.2.3. Test parameters a) test repeated times b) test frame length c) test frame rate 6.2.4. Test process 1. Configure the VM tester to offer traffic to the V-Switch with the input frame changing from the maximum rate to the rate with no frame loss at reducing 10% intervals according to RFC 2544. 2. Record the input frame count and output count on VSwitch. 3. Calculate the frame loss percentage under different frame rate. 4. Change the frame length and repeat from step1 to step4. Huang, et al. Expires January 18, 2018 [Page 9] Internet-Draft virtual-network-performance-benchmark-03 July 2017 6.2.5. Test result format ----------------------------------------------------------------- |Byte|Maxmum rate |90% Maximum |80% Maximum |...| rate with | | | (Gbps) | rate (Gbps)| rate (Gbps)| | no loss (Gbps)| ----------------------------------------------------------------- | 64| | | | | | ----------------------------------------------------------------- | 128| | | | | | ----------------------------------------------------------------- | 256| | | | | | ----------------------------------------------------------------- | 512| | | | | | ----------------------------------------------------------------- |1024| | | | | | ----------------------------------------------------------------- |1518| | | | | | ----------------------------------------------------------------- Figure 6: test result format 6.3. CPU consumption The objective of the test is to determine the CPU load of DUT(VSWITCH). The operation of DUT (VSWITCH) can increase the CPU load of host server. Different V-Switches have different CPU occupation. This can be an important indicator in benchmarking the virtual network performance. 6.3.1. Objectives The objective of this test is to verify the CPU consumption caused by the DUT (VSWITCH). 6.3.2. Configuration parameters Network parameters should be defined as follows: a) the number of virtual tester (VMs) b) the number of vNIC of virtual tester c) the CPU type of the server d) vCPU allocated for virtual tester (VMs) e) memory allocated for virtual tester (VMs) Huang, et al. Expires January 18, 2018 [Page 10] Internet-Draft virtual-network-performance-benchmark-03 July 2017 f) the number and rate of server NIC 6.3.3. Test parameters a) test repeated times b) test frame length c) traffic rate 6.3.4. Test process 1. Configure the VM tester to offer traffic to the V-Switch with certain traffic rate. The traffic rate could be different ratio of NIC's speed. 2. Record vSwitch's CPU usage on the host OS if no packets loss happens. 3. Change the traffic rate and repeat from step1 to step2. 4. Change the frame length and repeat from step1 to step3. 6.3.5. Test result format Huang, et al. Expires January 18, 2018 [Page 11] Internet-Draft virtual-network-performance-benchmark-03 July 2017 -------------------------------------------------- | Byte| Traffic Rate(Gbps)| CPU usage of vSwitch | -------------------------------------------------- | | 50% of NIC speed | | | |------------------------------------------- | 64 | 75% | | | |------------------------------------------- | | 90% | | -------------------------------------------------- | | 50% of NIC speed | | | |------------------------------------------- | 128 | 75% | | | |------------------------------------------- | | 90% | | -------------------------------------------------- ~ ~ ~ ~ -------------------------------------------------- | | 50% of NIC speed | | | |------------------------------------------- |1500 | 75% | | | |------------------------------------------- | | 90% | | -------------------------------------------------- Figure 7: test result format 6.4. MEM consumption The objective of the test is to determine the Memory load of DUT(VSWITCH). The operation of DUT (VSWITCH) can increase the Memory load of host server. Different V-Switches have different memory occupation. This can be an important indicator in benchmarking the virtual network performance. 6.4.1. Objectives The objective of this test is to verify the memory consumption by the DUT (VSWITCH) on the Host server. 6.4.2. Configuration parameters Network parameters should be defined as follows: a) the number of virtual tester (VMs) b) the number of vNIC of virtual tester c) the CPU type of the server Huang, et al. Expires January 18, 2018 [Page 12] Internet-Draft virtual-network-performance-benchmark-03 July 2017 d) vCPU allocated for virtual tester (VMs) e) memory allocated for virtual tester (VMs) f) the number and rate of server NIC 6.4.3. Test parameters a) test repeated times b) test frame length 6.4.4. Test process 1. Configure the VM tester to offer traffic to the V-Switch with certain traffic rate. The traffic rate could be different ratio of NIC's speed. 2. Record vSwitch's MEM usage on the host OS if no packets loss happens. 3. Change the traffic rate and repeat from step1 to step2. 4. Change the frame length and repeat from step1 to step3. 6.4.5. Test result format Huang, et al. Expires January 18, 2018 [Page 13] Internet-Draft virtual-network-performance-benchmark-03 July 2017 -------------------------------------------------- | Byte| Traffic Rate(Gbps)| MEM usage of vSwitch | -------------------------------------------------- | | 50% of NIC speed | | | |------------------------------------------- | 64 | 75% | | | |------------------------------------------- | | 90% | | -------------------------------------------------- | | 50% of NIC speed | | | |------------------------------------------- | 128 | 75% | | | |------------------------------------------- | | 90% | | -------------------------------------------------- ~ ~ ~ ~ -------------------------------------------------- | | 50% of NIC speed | | | |------------------------------------------- |1500 | 75% | | | |------------------------------------------- | | 90% | | -------------------------------------------------- Figure 8: test result format 6.5. Latency Physical tester's time refers from its own clock or other time source, such as GPS, which can achieve the accuracy of 10ns. While in virtual network circumstances, the virtual tester gets its reference time from the clock of Linux systems and it's hard to make the physical and virtual tester keep precisely synchronized. So We use the traffic model B as the latency test model. ______________________________________________________________ | | | ---------------- ---------------- ---------------- | | |Virtual tester|----|DUT (VSWITCH) |-----|Virtual tester| | | ---------------- ---------------- ---------------- | | Server | |______________________________________________________________| Figure 9: time delay test model Huang, et al. Expires January 18, 2018 [Page 14] Internet-Draft virtual-network-performance-benchmark-03 July 2017 6.5.1. Objectives The objective of this test is to verify the DUT (VSWITCH) for latency of the flow. This can be an important indicator in benchmarking the virtual network performance. 6.5.2. Configuration parameters Network parameters should be defined as follows: a) the number of virtual tester (VMs) b) the number of vNIC of virtual tester c) the CPU type of the server d) vCPU allocated for virtual tester (VMs) e) memory allocated for virtual tester (VMs) f) the number and rate of server NIC 6.5.3. Test parameters a) test repeated times b) test frame length 6.5.4. Test process 1. Configure the virtual tester to offer traffic to the V-Switch with the traffic value of throughput tested in 6.1. 2. Keep the traffic for a while and then stop it, record the minimum, maximum and average latency. 3. Change the frame length and repeat from step1 to step4. 6.5.5. Test result format Huang, et al. Expires January 18, 2018 [Page 15] Internet-Draft virtual-network-performance-benchmark-03 July 2017 +-----+-----------------+-----------------+------------------+ | Byte| Min Latency | Max Latency | Average Latency | +-----+-----------------+-----------------+------------------+ | 64 | | | | +-----+-----------------+-----------------+------------------+ | 128 | | | | +-----+-----------------+-----------------+------------------+ | 256 | | | | +-----+-----------------+-----------------+------------------+ | 512 | | | | +-----+-----------------+-----------------+------------------+ | 1024| | | | +-----+-----------------+-----------------+------------------+ | 1518| | | | +-----+-----------------+-----------------+------------------+ Figure 10: test result format 7. Security Considerations None. 8. IANA Considerations None. 9. Normative References [RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC2234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, DOI 10.17487/RFC2234, November 1997, . [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, . Huang, et al. Expires January 18, 2018 [Page 16] Internet-Draft virtual-network-performance-benchmark-03 July 2017 Authors' Addresses Lu Huang (editor) China Mobile 32 Xuanwumen West Ave, Xicheng District Beijing 100053 China Email: hlisname@yahoo.com Rong Gu (editor) China Mobile 32 Xuanwumen West Ave, Xicheng District Beijing 100053 China Email: gurong@chinamobile.com Bob Mandeville Iometrix 3600 Fillmore Street Suite 409 San Francisco, CA 94123 USA Email: bob@iometrix.com Brooks Hickman Spirent Communications 1325 Borregas Ave Sunnyvale, CA 94089 USA Email: Brooks.Hickman@spirent.com Huang, et al. Expires January 18, 2018 [Page 17]