Certess tool tests verification environments

By Michael Santarini, Senior Editor -- 5/8/2007

EDA start-up Certess Inc has announced the Certitude tool, which uses a mutation-analysis software technique to help designers locate improperly functioning areas in their verification flows. The “functional-qualification tool is to verification what verification is to design, meaning that it verifies the verification,” says Michel Courtoy, chief executive officer of Certess. Mark Hampton, chief technology officer of Certess, says that a traditional verification environment comprises stimuli that simultaneously activate the design under test and a reference model. After it activates or exercises a problematic region of the design, it then propagates that problem area to a detection engine. The detection engine then lets the user know that it has detected a bug.

“If you don’t activate the bug, if the bug doesn’t propagate, or if the detection mechanism doesn’t work, the bug goes to silicon,” says Hampton. “Code coverage and functional coverage focus only on measuring how good the stimuli are at activating or exercising a design’s behavior. Until now, verification engineers haven’t had any visibility into the ability of the verification environment to propagate or detect bugs. Functional qualification spans all three aspects of the verification process. It is able to measure the ability of the verification environment to activate areas of the design, the ability to propagate potential bugs, and the ability to check them. … It gives engineers visibility into the propagation and detection.”

“Mutation analysis has been around since 1978, but has mainly been a software research topic,” says Hampton. “Certess is the first company to apply this technology to industrial projects.” He notes that research has proved mutation analysis to be effective. However, the runtimes were too long for industrial applications. Now, Certess has built a new way of applying this analysis. Users feed their Verilog or VHDL into Certitude, and the tool introduces special types, faults, or mutations, in the HDL. If the verification environment can’t even detect the mutations, a real error in code would also go undetected. For example, the fault A=B and C may replace a small bit of your code that says A=B or C. If your verification environment is not detecting this error, it is probably also not activating, propagating, or detecting some more complex threads in your code. In addition to Verilog and VHDL files, users feed Certitude test-case IDs, as well as compile-and-execute scripts that control communication between Certitude and the environment under test. The tool then outputs an HTML report that tells you the location of an unpropagated or undetected mutation.

The company has more than 50 customers for the tool, including Juniper Networks and ST Microelectronics. The technology has been popular as a verification-improvement technology and as a verification-quality metric. As a verification improvement technology, engineering teams use the tool to find weaknesses in their verification environments. If you have a fault in your design, the tool propagates to the outputs of the design through the design and shows where you injected the mutation so that you can analyze it to see what it is missing. You then must either add or fix the checker. If the mutation doesn’t affect the behavior of the design and the test part, it probably indicates a condition the checker is missing or the existence of a bug or error in one of the checkers, Hampton explains.

As a verification-quality metric, Certitude uses a statistical sampling technique. You might, for example, have an IP (intellectual-property) block, run the metric on it, and receive a result of 87%±2%. That result means that Certitude can detect that percentage of mutations in your design. You can then run the same block on different verification environments or different IP blocks on the same environment to pinpoint the location of weaknesses in you flow. The tool currently supports Verilog and VHDL or mixed Verilog and VHDL designs. Hampton notes that the mutation-analysis technology is not language-dependant so Certess also plans to expand it for assessing higher level verification environments, such as SystemC environments or environments that employ emulation. He notes the tool would be ineffective for mixed-signal or analog design, because many more variables could affect those designs. Certess offers Certitude with a starting price of $100,000 for a one-year subscription.

© 2009, Reed Business Information, a division of Reed Elsevier Inc. All Rights Reserved.