A Concrete Linux Virtual Platform Example

Virtual platforms are used to find many different types of system and software issues. Of course, platforms take some time to develop and debug (regardless of what you read in marketing brochures), but in most situations the benefits outweigh the time and effort required for platform creation. Generally, I think of platform creation as a verification activity in the same category as a hardware testbench. If a design has no bugs, then there is no need to spend time developing a testbench or doing verification. Most engineers know this is not reality, and verification is generally time well spent. For readers who haven't used simulation to run embedded software, it's good to communicate concrete scenarios that make concepts more real.

Not too long ago, I downloaded the latest kernel for the Xilinx Zynq-7000 Programmable SoC to make sure it was running smoothly on the Cadence virtual platform. Surprisingly, I found it was not. It ran fine if I booted from a RAM-based file system, but when I tried to run Ubuntu 12.10 from SD card, it didn't boot. Debugging the hang pointed to something wrong with the UART. Strangely, I couldn't imagine how changing the boot media had anything to do with the UART. Being a virtual platform developer it can be difficult to determine if a hang during Linux boot is related to the UART driver or the UART model. Since I couldn't tell, I changed the UART model to make the system run and continued with life. Some days later, I downloaded the latest Zynq Linux again and spotted a checkin that looked like a fix for my previous UART problem. However, when running the new version I was surprised to see a warning that the software had accessed a bad address in the UART register map.

After some more digging I found the fix was indeed for my hang, but it accidentally inserted a new bug into the UART driver. The inserted bug reversed the arguments on the function xuartps_writel() which is used to write a UART data character to the UART FIFO.  The arguments should be data followed by the address, but the bug had the address first, then the data. The result was a write to a UART offset address that was really the data. If the data value was a lower case ‘c' which is 0x63 in hex, then a write would occur to address offset 0x63. Since this is not a real address in the UART, the software engineer may not even notice that the character didn't appear on the screen and the system may continue to run as normal.

With a Virtual Platform these situations are easier to catch. All generated SystemC peripheral models flag accesses to undefined register offsets. The system router will also catch accesses that are beyond the defined size of the memory space of a peripheral model. Below is a screenshot of the error detected by the Virtual Platform.

In addition to identifying problems, virtual platforms can also stop immediately and the software debugger will point at the line of code that caused the problem. This avoids hunting through the Linux source code to find where the bad hardware access may be coming from. A screenshot of showing the highlighted line in the source code is below.

The story demonstrates how software bugs can accidently be inserted, but quickly identified using a simulator. When running on real hardware the bad register accesses may or may not be noticeable or cause any problem. The commit message mentions that the problem only occurs "under certain circumstances" so it wasn't as if the UART was completely broken. Virtual platforms can also be a great way to automate testing of new Linux kernels to avoid any regression as code is added and changed.

Jason Andrews