Today’s guest blog is Rob Dekker, the President and principal developer at Verific, who produce front-ends used by many EDA tool suppliers both big and small. Prior to founding Verific, Rob was at Exemplar where he was the principal developer of their Leonardo FPGA synthesis product (which was acquired by Mentor in 1995). He started his career at Philips Research in the Netherlands after graduating from Delft University of Technology.
SystemVerilog and the triangle of truth
With the advent of the IEEE 1800-2009 SystemVerilog standard, currently in the proposal stage, we thought it would be a fun exercise to take stock of support for the 1800-2005 version of this mighty hardware description language (HDL). For this purpose, we compared the two leading SystemVerilog simulators Questa (from Mentor Graphics) and VCS (from Synopsys) with our own Verific SystemVerilog parser, a front end widely used in EDA applications marketed by our customers.
As an aside for those not familiar with Verific Design Automation, we build VHDL, Verilog, and SystemVerilog parsers, analyzers and elaborators, and license those to a wide variety of semiconductor, FPGA and EDA companies. A representative list of our customers can be found on our website (www.verific.com) and there is no need to bore the reader with those here. If you do RTL design, chances are you have used one of our parsers. We never disclose which components our customers’ license from us, but we have our fair share of SystemVerilog licensees and carry the battle scars to prove it.
We also have a SystemVerilog test suite that we run internally to validate our own results, and several of our customers have licensed this test suite as well. It contains 2,500 tests with positive tests (“this should pass”) and negative tests (“this should fail”). The majority of these are semantics tests, and 40% is synthesizable. By no means do we claim that this test suite is complete or even sufficient. As a matter of fact, its size is dwarfed by our Verilog 2001 and VHDL regression suites. We are sure that over time we will be able to extend our test suite as more SystemVerilog design activity starts occurring in the industry.
Our experiment was simple. We took the tests from our SystemVerilog test suite and ran them simultaneously with VCS version 2008.12, Questa version 6.5a, and Verific 2009.01. We aptly named it the “Triangle of Truth” and categorized the results in eight bins ranging from “0 0 0” (all three tools parse and report the same error on the test) through “1 1 1” (all three tools parse and pass the test). The good news is that in 73% of the tests, all three parsers fully agreed. The bad news, of course, is that there are discrepancies in the remaining 27%.
The bin where Questa and VCS both successfully parsed the test but Verific failed were considered defects and quickly fixed in the Verific parser. There is also a category where both VCS and Questa reported an error in the design as expected but where Verific failed to flag the error. These results are called “false positives” and require improved semantic checking in the Verific parser. They are less onerous but should get fixed over time as well.
And, then there were situations were either Verific and Questa agreed, or Verific and VCS agreed, but not Questa and VCS. Remember the saying about the person with two watches who never knows what time it is? Well, that’s how we feel in situations like this. Predictably, Questa customers nudge us toward their simulator’s implementation and VCS users like it exactly the other way around. Where does that put us? Between a rock and a hard place!
Worse yet are those situations where one of those leading EDA tools clearly violates the standard. What are the others to do? Fortunately, that happens only rarely. Conflicts between the three vertices of the “Triangle of Truth” mainly relate to design checking, where tool A issues an error while tool B parses the same design just fine, or vice versa. It will be interesting to see what the 1800-2009 version of the SystemVerilog standard will do to the “Triangle of Truth.” Improved clarity of the standard should make it easier for EDA tools to agree on interpretation, but additions to the language will undoubtedly generate their own category of mismatches. Whichever way it goes, the “Triangle of Truth” will remain a useful tool to measure SystemVerilog interoperability.