Moving to virtual emulation for software-defined networking

-August 18, 2017

Tremendous shifts in networking research and development over the last few years have resulted in a whole new class of networking devices, more widely known as Software Defined Networking (SDN) switches and routers. Developments using Reconfigurable Match Tables (RMT) and other Reduced Instruction Set Computing (RISC) style architectural choices have transitioned network switch System on a Chip (SoC) designs from fixed-function topologies into highly configurable match action devices.

In order to operate, an SDN device must be managed using a SW application that downloads its configuration. This is done most commonly using a Peripheral Component Interconnect Express (PCIe) interface. Networking SoC verification has traditionally relied on PCIe as an essential interface for simple device access, configuration and runtime case management. PCIe has been the dominant networking SoC management channel of choice for decades.

However, SDN SoCs introduce significantly greater interaction between the host application SW state and the SoC HW state being managed over PCIe. SW reconfigurations are rapid, frequent and can support thousands of individual switching profiles. HW and SW co-verification has emerged as a must have methodology in order to satisfy these larger validation objectives.

Traditional PCIe verification methodologies, in particular vector-based verification methodologies, are insufficient for big device co-verification. Whereas vector-based verification methodologies are moderately suitable for SW configuration, they have very poor suitability for HW/SW co-verification. Consequently, they simply do not work for SDN.

The best way to overcome their limitations is to shift to a virtual paradigm using virtual PCIe. As we will show, virtual PCIe is the right tool for SDN, accerlerating the move to virtual emulation for developers working in this space.


SDN technology overview

In a nutshell, SDN separates the control and forwarding functions of a network switch. It centralizes control of the network to a software application on an external host. For SDN devices, the separation of control and forwarding functions is distinct. Centralization of the control and the ability to program the behavior of the network using well-defined interfaces is a key hallmark of the SDN paradigm.

Figure 1
Example SDN HW Block Diagram [Cavium, “XPliant Ethernet Switch Family”]

A key component is the SW used to configure the elements shown in blue. Essentially the chip wakes up not knowing how to do anything. SDN SoCs are not traditional fixed function devices, but instead act as highly configurable networking data plane devices that are programmed from what is typically referred to as an SDN Control Plane Application or Orchestrator. The orchestrator handles everything from initial device configuration to real time control. It can manage a single SoC or a mesh of devices. Here “managed” means that a processor gets involved, unlike an unmanaged switch where no processor is coupled with the networking device.


Virtual methodology

Virtual PCIe enables applications to interact with the emulation DUT just as if it there were real silicon sitting on the bench, making it the ideal methodology for HW/SW co-verification. It is noteworthy that the SDN device operates at a much slower speed in emulation as compared to targeted product wall clock speed. However, from the DUT’s perspective, it is running at the full targeted rate because all clock ratios are maintained through the PCIe interface and DUT emulation clocks.

An example topology is shown in Figure 2, where a virtual machine (VM), in this case Quick EMUlator (QEMU), runs on a host machine that is hosting the Red Hat or SuSE OS. A guest Ubuntu OS is launched inside QEMU, which then loads the appropriate kernel modules necessary to run the SDN SoC PCIe driver. This driver manages the SDN HW DUT behind PCIe3.


Figure 2
Virtual PCIe3 Control and Data Path Overview

The SystemC Proxy and HDL Transactor handles are exchanged between the VM and emulator under the hood. The QEMU VM can also be physically run on the SDN software development kit (SDK) hardware. A virtual PCIe root complex supported by Mentor is delivered in a solutions bundle called Veloce® VirtuaLAB. The library supports virtual machine types for many technologies, such as networking, multimedia, storage, automotive, CPU, etc.

When using virtual PCIe, the PCIe software under test driver is the exact same driver that customers would receive. The driver enumerates the bus and finds the SDN DUT just like the real product would. SDN SW seamlessly connects through the PCIe root complex with the SDN SoC end point (EP) to initialize the DUT and configure the host SW state dynamically. Obviously this is very appealing to both SW and HW developers during design verification, laboratory validation and customer support.

Using virtual PCIe breaks down the walls between pre and post silicon. Take the case where 100s of tests must be run on 100s of devices before such samples can be shipped to customers after fab out. Tests for functional, device-yield-voltage-temperature, and SDN gold standard applications all can be run immediately on each sample, even before the automated test and measurement is available. All tests validated during pre-silicon development can be leveraged into post-silicon testing because they use the logically identical platforms under the virtual PCIe VM paradigm.

There may be cases where applications running in the QEMU VM need to be slowed down to better match the actual wall clock rate of the emulation DUT. In this way, built-in SW timeouts within applications can be satisfied if present. If the targeted applications have no timeout requirements, those applications can be run at the default VM clock rate in QEMU w.r.t. to the DUT emulation clock.

Once design teams become aware of the advanced features VM provides, and given a choice, these teams usually choose to adopt virtual PCIe to develop product SW and drivers in parallel with the HW development cycle. SW and HW architectures can be co-verified and confirmed early in the development cycle, requiring less code rip up. Functional SW APIs, which were impossible to test using A vector-based verification, can now be tested using virtual PCIe. High Availability (HA) APIs are such an example.

An HA API must program devices on the fly while traffic is streaming, in order to verify that the API works as expected under all traffic scenarios, maintenance, and parallel processes. Operators can shutdown an HA API server for maintenance. A secondary CPU picks up the SDN state before the primary CPU powers down. Once maintenance completes, the primary management CPU reboots and reruns the entire initialization sequence without altering any HW state. The management CPU rebuilds its SDN SW state from the residual state found in the still running SDN HW. Without virtual PCIe testing, such SW/HW interactions are literally impossible to validate pre-silicon.

Testing all SW state space in relation to HW is a tremendously large and complex task. For this and many other verification objectives highlighted in the introduction section, enabling a DUT’s reactive configuration through virtual PCIe becomes a mandatory test capability for development teams that see the value in sequencing through the entire product code space before tape out.


Performance considerations

Depending on how SDK SW is written, it may not function as a typical PCIe application might. For example, an SDK which uses the host CPU to directly RD/WR (PIO) large images into an SDN Ethernet design may have poorer performance than a SW architecture that uses endpoint initiated RD/WR (DMA) to transfer large amounts of data. For example, some customers may decide to simplifiy SW and use only PIO instead of EP DMA, at the sacrifice of some management channel performance.

End point initiated DMA has many advantages: the CPU isn’t loaded because it only talks to local memory as required, and any transport latencies can be hidden by making multiple read requests by the EP, (CPUs doing PIO cannot typically hide such latencies, especially for RD).

Regardless of EP optimization over large PCIe image load processes, the real aim for virtual Ethernet plus virtual PCIe is to verify large SDN designs in which emulation time for SDN Ethernet testing is presumably much greater than PCIe configuration download times for most test scenarios. This is important because it highlights the distinction between Mentor’s Out Of the Blue (OOB) transactor methodology versus simulation-ratio (sim-ratio) models when arbitrating between multiple VMs, as is the case with PCIe and Ethernet VMs on the same host workstation.

As expected, when using OOB the PCIe performance is not throttled. It will always be maximally aggressive, so when Ethernet and PCIe are sharing the same co-model channel, an OOB PCIe VM will almost certainly affect the wall clock Ethernet VM performance more than the sim-ratio between VMs. Notice in Figure 3 that this could happen if both virtual Ethernet and virtual PCIe are running using the same co-model channel to communicate with the DUT at the same time.


Figure 3 Virtual Ethernet Device Architecture

However, PCIe and Ethernet are usually running exclusive of each other. When PCIe is running, it’s required to be fast for configuration, and we’re not so interested in the Ethernet wall clock performance while configuring the chip. OOB performance has been optimized for virtual PCIe, but this has not been the essential metric for most customers. Their primary interests have been the many key features that virtual PCIe enables, such as checkpoint save/restore, protocol analyzer, flexibility and advanced debug. For many customers the performance hit in moving from ICE to virtual is in the noise compared to what these key features provide in SW/HW co-verification.

When migrating from ICE to virtual emulation, one customer migration showed an ICE baseline of 10 minutes for chip initialization; whereas virtual PCIe took 14 minutes to complete the same initialization and boot. However, when using the checkpoint save/restore feature with Veloce VirtuaLAB, the solution was able to configure the chip in just 4 minutes, thereby turning a 4 minute slow down into a 6 minute speed up when using an initialization checkpoint image.

To get a relative sense of virtual time versus a real-time wall clock metric between emulation and a device on the bench, one customer example showed 15 seconds of SDK real time equated to approximately 30 minutes of emulation time using the PCIe VM implementation shown in Figure 2.


Debug and ease of use

A key differentiator in any platform used for validation is the ease of debugging difficult problems. A corollary to that goal is to have as few unique verification platform types as possible, in order to reduce the development and maintenance load on HW and SW teams pre and post-silicon validation. Virtual PCIe can be used pre and post-silicon. Virtual PCIe enjoys the advantage of having all transactions between host and emulator existing as 100% platform SW. Therefore all transaction are 100% visible.

In general, the VM paradigm enables improved debugging capabilities. However, Veloce VirtuaLAB goes several steps beyond this by providing a dedicated protocol analyzer application that looks very much like a LeCroy PCIe analyzer. For example, shown in Figure 4, notice the application disassembles all Data Link Layer Packets (DLLP) and Transaction Layer Packets (TLP), as well as supports many other statistics and tracing capabilities. A detailed GUI front end organizes this information into a single utility that has access across the entire PCIe stack. The analyzer is coupled directly to the VM PCIe stack and therefore provides extremely fast Time-To-Visibility (TTV) on all trace and analysis.

Figure 4 Virtual PCIe Protocol Analyzer

SW Checkpoint and restore

SW checkpoint and restore is another key differentiator for VM emulation platforms. Virtual PCIe and virtual Ethernet support SW checkpoint and restore, but VTL does not because the user’s testbench is essentially a custom environment that emulation tools would not necessarily know how to checkpoint. A vector-based verification methodology using VTL is essentially a set of custom APIs using C++ or UVM with a simulator engine running the bench.

The effort to make users’ testbenches checkpoint compatible is something most organizations have not been willing to invest in doing. SystemC/C++ and UVM vector-based verification are essentially very different use models compared to that of virtual PCIe, and therefore true SW checkpoints have not been needed. Repeatability using traditional test bench and emulation replay mechanisms have been satisfactory for the vector-based verification paradigms. That said, VTL is always the right tool to use when connecting a testbench to a PCIe emulation target.


Virtual PCIe delivers a “shift left” in software defined networking emulation. Virtual PCIe and Ethernet virtual machines address the big device management channel requirements for HW and SW co-verification in emulation, with virtual machines achieving greater functional coverage. Advanced development considerations highlight how checkpoint save/restore, protocol analyzer, SW configuration flexibility, ease of use and advanced debug features are markedly improved and combined into a single virtualization methodology platform.

Virtual PCIe is the right tool for SDN, accelerating the move to virtual emulation for developers working in this space. It enables applications to interact with the emulation DUT just as if it were real silicon sitting on the bench, making it the ideal methodology for HW/SW co-verification.

For more details on SDN and virtual PCIe, please read the new whitepaper Virtual PCIe Delivers a “Shift Left” in Software Defined Networking Emulation


Also see:

Ron Squiers is a networking specialist who received his BSEE from the University of California Berkeley and did graduate studies at Stanford University. Ron has several patents in the networking, encryption and emulation domains. He has developed next generation networking, communications and encryption products for Mentor Graphics, Broadcom, HP, SGI, & AMD.




Loading comments...

Write a Comment

To comment please Log In