Embedded Software Bugging and Debugging

In one of my previous posts I introduced an interesting book titled Dreaming in Code. One of the great quotes from the book pertains to today's subject, embedded software debugging. If you don't have the book you can look it up on the book excerpt page of the author's website for the book.

If you talk with programmers about this, prepare for whiplash. On the one hand, you may hear that things have never looked brighter: We have better tools, better testing, better languages, and better methods than ever before! On the other hand, you will also hear that we haven’t really made much headway since the dawn of the computer era. In his memoirs, computing pioneer Maurice Wilkes wrote of the moment in 1949 when, hauling punch cards up the stairs to a primitive computer called EDSAC in Cambridge, England, he saw the future: “The realization came over me with full force that a good part of the remainder of my life was going to be spent in finding errors in my own programs.” From Wilkes’s epiphany to the present, despite a host of innovations, programmers have been stuck with the hard slog of debugging. Their work is one percent inspiration, the rest sweat-drenched detective work; their products are never finished or perfect, just varying degrees of “less broken.”

An article I read today from EDN titled "Shedding light on embedded debugging" confirmed that the situation has not changed much. For the last three years about one third of the respondents identified debugging as the number one thing they would like to improve about embedded software design. The article covers a wide range of debugging technologies that are available such as virtual system prototypes, better host tools, on-chip debugging features, and trace buffers. Even with all this available the survey indicates that all of these have not been able to make a dent in the number of engineers identifying debugging as the main thing they would like to improve.

One of the items discussed in the article is that many embedded systems don't lend themselves to debugging since they contain hardware like sensors and actuators that are hard to model and make traditional breakpoint based debugging difficult. I don't recall who told me this info, but about 12 years ago Cygnus (long since acquired by Red Hat) was promoting something in gdb called tracepoints because some engineers had to debug code that was controlling an elevator. Setting a breakpoint in the software as the elevator was on the way up wasn't a good idea as it may cause the elevator to blow out through the top of the building. I don't know if the feature is still in use, but it's still in gdb, just type "help tracepoints" at the gdb prompt.

Unfortunately, I don't have the solution to all the world's debugging problems, but I do have a very small feature that may be of interest to some of you. The feature is part of ISX and is called Embedded Software Trace. It's a plugin for SimVision that helps figure out what embedded software is doing during verification. Like the elevator problem, many engineers doing SoC verification have to deal with software and find that interactive debugging is not always practical in a batch and farm environment. Embedded Software Trace links the hardware debugging environment in SimVision with what is happing in the software running on a model of the CPU in the simulation.

The user can move a cursor as normal on the SimVision waveform (or any other SimVision window) and the Source Browser will display the location of the executing software. The Embedded Software Trace toolbar on the Source Browser has some nifty features like next and step (forward and backward) and search for addresses, functions, and combinations of source file and line number. This makes it very easy to find out what is happening in the software at the same time you are debugging the hardware. Below is a screen shot of a Source Browser with the Embedded Software Trace toolbar. The screen shot is also a sneak preview of SimVision 8.2 as regular SimVision users will notice the completely new look and feel for 8.2 that will be released in Q4.

Embedded Software Trace is just another SimVision window type, and you can find it by doing Windows -> New -> Embedded Software Trace or type "window new eswtrace" at the SimVision> prompt. Of course, you need to have Specman installed since ISX is delivered as part of Specman.

One of the great benefits of blogging is the ability to get information about these nifty features directly into the hands of people that need them. When I was working in small companies it was easy to promote such things to users, but in a company the size of Cadence this is a small feature of ISX, and ISX is part of an ESL package, and the ESL package works with the Incisive Simulator, and the Incisive Simulator is actually something users can buy. As the R&D project leader for ISX, blogging is truly the best way to communicate things like Embedded Software Trace to users.

I hope you like it and feel free to send feedback and ideas on how to improve things.