The state of SoC verification: a morning with Mentor's John Lenyo

Every once in a while it’s a good idea to step back from the day-to-day routine of product announcements and take stock of just where we stand on an issue. I had a chance to do that last week with John Lenyo, general manager of the Design Verification and Test division at Mentor Graphics. Mentor has its own commercial axes to grind, of course, and Lenyo said that in the verification space the company is focused on the largest SoCs, not on smaller or trailing-edge chips. So there is a bias to his data. But that said, the executive is in a great position to collect observations from across the industry.

His first observation is that at the most abstract level, not much has changed. "The verification problem is still growing geometrically, not linearly, with design complexity," Lenyo said. "But we are getting no relief either from schedules or, these days, budgets. So verification engineers must continually grow more productive."

To complicate matters, growing complexity is not the only problem. In recent years SoCs have included analog and mixed-signal circuitry with increasing frequency. Now the boundaries between the digital and analog blocks are eroding, creating the need for very agile mixed-mode verification. And then there is power. "With every new process node, designers have to come up with a whole new set of techniques for reducing power," Lenyo said. "And every one of these new techniques creates a new verification problem." Indeed, some managers today are saying that not only have they had to add complex power-controllers to their designs, but that the complexity of the power management has forced them to treat clock and supply nets as signal nets in the verification process.

In this context, getting more productive is not easy. Most of the ideas that have allowed verification teams to scale in the past are running out of steam, according to Lenyo, and while there are promising new ideas, there are no silver bullets on the horizon. The good news for productivity is that the Open Verification Methodology is catching on. Downloads of the base class libraries to support this verification approach continue to mount, and the community continues to grow. OVM is Mentor-Cadence-oriented and built on SystemVerilog, so it’s not the only game out there. But it is an indicator.

In the general market, Lenyo said, about 40 percent of designs have some meaningful level of assertions written for them. So the whole concept of reusable verification IP, embedded assertions in a standard language, and systematic verification is catching on. There is even momentum for SystemVerilog as a standard language for test benches. In a Mentor study, C/C++ is still the most widely-used language for test benches, as it is for behavioral-level design description. But SystemVerilog has already passed VHDL in frequency of use, and is growing rapidly, while the use of C/C++ is shrinking.

But fundamentally, Lenyo admitted, verification methodology is still rooted in functional verification using simulation tools. The productivity improvements have come not from replacing simulation with some magic new concept, but by finding ways to make it more productive. And even as some of these ideas are just getting off the ground in the industry mainstream, they are running out of steam in high-end designs, Lenyo worried.

Part of the problem is that simulation with constrained random vectors—the primary tool for many test benches today—cannot scale with increasing complexity of design. It’s just not enough any more. On one hand, it’s become a superhuman task to estimate the degree of design coverage from a huge number of constrained random tests devised in different places by different parts of the design and verification teams. On the other, the sheer volume of cycles necessary to execute the tests is running into the limitations of server farms. "AMD has gone from 50 CPUs to 10,000 CPUs for verification in the course of a few years," Lenyo remarked. That kind of growth is not sustainable, especially if it is geometric. Even worse, the total size of designs is now bumping into the limitation of servers’ 64-bit address space. It may no longer be possible to load a complete design without partitioning it, let alone run regressions on it.

Design teams have tried two stratagems to deal with the growth in complexity, and both of those are now losing their effectiveness, Lenyo said. One has been to try to escape part of the verification problem by licensing pre-verified IP. This has worked poorly. One manager Lenyo cites has finally begun to assume that all incoming IP is broken until proved otherwise.

Another tactic has been to outsource some or all of the verification workload. This doesn’t solve the scaling problem, but ideally it passes the problem on to another venture with, one hopes, lower labor costs and more servers. "But now it’s the contractor who needs to increase productivity. The problem hasn’t gone away," Lenyo said.

So what are teams doing to escape the verification bind? The first thing Lenyo mentioned is systematic, quantitative verification management, based on coverage metrics and centralized tracking. "In a survey we did recently, the most frequently reported concern of chip-design managers was coverage closure," Lenyo said. "It has replaced timing closure as the biggest worry."

Part of the problem is that there is no single metric. Bug-tracking—watching the number of new bugs decline with time—is now pretty much discredited as a metric. Code coverage, functional coverage, and assertion coverage all play roles, but in the end there is no quantitative answer. Managers can only prepare a verification plan at the outset, and then attempt to measure coverage progress against the plan. To this end Mentor and other vendors have come up with unified coverage databases that allow managers to at least see where the verification engineers think they are in the process. This capability has several benefits, according to Lenyo. Not only does it allow managers to direct their resources to the least-covered portions of the design, but it allows them to avoid redundant testing. "One user has been able to reduce their regression-test run time by 60 percent just by not running unnecessary tests," Lenyo said.

Along with better management tools, many teams have added formal verification to their flows—not as the panacea that it was once hoped to be, but as a very valuable point tool. "One big win for formal tools has been in clock-domain crossing analysis," Lenyo said. "Just about everywhere people are using those tools." Two other areas are final quality-assurance, where verification engineers sign off small blocks of the design against a set of formally testable assertions, and—perhaps surprisingly—debug.

"Debug is an interesting issue," Lenyo said. "It turns out that maybe half of the total verification time on a complex chip isn’t in actual verification, it’s in debug: trying to understand why something didn’t work." And it turns out that formal verification, with its ability to trace a particular behavior back to its causes over a small portion of a logic design, can be an ideal tool to aid debug—if that tool is interactive and has a debug-oriented, rather than a mathematician-friendly, user interface. This has led to the emergence of a new category of formal tools in recent years—a particular interest of Mentor’s.

Another strategy for trying to get leverage against the growing problem has been exploiting multicore servers and multiprocessor parallelism. In some ways, this is natural. "Test benches are inherently parallel," Lenyo observed, pointing out that you can run different tests on different CPUs without any partitioning or software design work whatsoever. "But after a while, you run out of test bench parallelism and still have idle CPUs." So verification tool vendors are turning to the harder problem: finding parallelism not in the test bench, but in the design under test. This means partitioning the design along boundaries with the lowest frequency of signal crossings, and then assigning portions of the design to CPUs so that the rate of event-sharing between segments of the design pretty well lines up with the bandwidth between the CPUs. In this way a verification run can use a cluster of multicore servers effectively. Mentor, for example, has shown sub-linear but still useful speed-up partitioning a design across 64 processors.

Finally, to deal with the power-management verification problem the industry is evolving a whole new set of tools to verify power integrity and power-management protocols. "We used to verify power management at the gate level," Lenyo reflected. "But customers started telling us the problem was too large to deal with at that level, and we had to verify power management at RTL. So we have moved in that direction."

Standards, especially including the emergence of SystemVerilog for test benches and assertions, systematic quantitative coverage management, and new point tools will all help. But so will just running simulation faster. And for that, there is no substitute for hardware emulation, Lenyo said. Consequently, the large-SoC teams Mentor tracks are increasingly turning to emulation—and the larger teams, Lenyo claims, tend to use purpose-build hardware emulation systems with their extensive user-support software rather than FPGA boards—to reduce simulation times. Not only is this a way of just reducing the time to run a given number of cycles, but it’s indispensable for situations where the stimulus to the design is a large data flow, like a video clip or a wireless waveform. Lenyo believes we will see more widespread use of the big, expensive emulation systems in the future.

So the problem continues to grow geometrically. We continue to allow insufficient time and budget for verification. And there are specific approaches to isolate and shrink some portions of the metastasis. But there is as yet no silver bullet, and it may be provable that there will never be one. That is not great news for verification managers, but it may be music to the ears of the EDA vendors.