Exploring the Virtual Platform Part 3

Welcome to part 3 of the "Exploring the Virtual Platform" series. For readers just joining please refer to Part 1 and Part 2 of the series to get up to speed.

Today's topic is debugging. One of the great things about a Virtual Platform is the ease of debugging code running on the platform. In the first 2 segments we demonstrated how to compile the Linux kernel and run it on the ARM Integrator platform. As I wrote about last year, step 1 is to run, and step 2 is to debug.

A typical task for a software engineer might be to add a new device driver to the kernel for a custom hardware peripheral. To get a feel for some of the ways used to debug drivers I highly recommend the book Linux Device Drivers. The 3rd edition is available for free as a set of pdf files. Chapter 4 is all about debugging. Some of the ways to debug include:

• Modifying the kernel configuration to output more debugging information
• Using printk()
• Using the /proc file system to export driver related information
• Calling ioctl() to the driver
• Running strace to trace system calls
• Using a debugger like gdb to debug the kernel

The last one is probably the most obvious, but doesn't seem to be commonly used. Chapter 4 of Linux Device Drivers states:

"Many readers may be wondering why the kernel does not have any more advanced
debugging features built into it. The answer, quite simply, is that Linus does not
believe in interactive debuggers. He fears that they lead to poor fixes, those which
patch up symptoms rather than addressing the real cause of problems. Thus, no
built-in debuggers."

Last year, I read an article that support for a debugger in the kernel may be coming, but I haven't followed up on it, mostly because with a Virtual Platform it's not necessary. Let's go back to the ARM Integrator now and see how to start debugging.

The first step is add some arguments to QEMU to tell it to wait for a gdb connection using -s and -S. More details of gdb usage can be found in the QEMU documentation.

jasona@hamlake:[arm-test]% qemu-system-arm -s -S -kernel ~/kernel/linux-2.6.17.1/arch/arm/boot/zImage -initrd arm_root.img

Now, QEMU is waiting for a gdb connection and it says  [Stopped] on the toolbar of the QEMU display.

Navigate to the place where the kernel was compiled, start the gdb from the Code Sourcery distribution on the file vmlinux (uncompressed kernel executable), and connect using the target remote command. A good place to put a break point is on start_kernel() and then hit c for continue. gdb will stop at start_kernel() and you are ready to debug the kernel.

jasona@hamlake:[linux-2.6.17.1]% arm-none-linux-gnueabi-gdb vmlinux
GNU gdb (Sourcery G++ Lite 2008q3-72) 6.8.50.20080821-cvs
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi".
For bug reporting instructions, please see:
<https://support.codesourcery.com/GNUToolchain/>...
(gdb) target remote :1234
Remote debugging using :1234
0x00000000 in ?? ()
(gdb) b start_kernel
Breakpoint 1 at 0xc000862c: file init/main.c, line 458.
(gdb) c
Continuing.

Breakpoint 1, start_kernel () at init/main.c:458
458             boot_cpu_init();
(gdb)
 

There are many interesting things that can be learned using the debugger to understand how the code works. The Integrator platform has 2 UARTs so experiment with breakpoints to see how the UART driver operates. gdb commands like

(gdb) info func uart*

will show all the UART related driver functions.  You can understand how the timer works by setting a breakpoint on integrator_timer_interrupt().

Notice the Virtual Platform is stopped when the debugger hits a breakpoint, this is a key benefit of a Virtual Platform, the debugging is completely non-intrusive. There is no need to connect a JTAG box to a board and scan commands in and out of a chip to control and inspect the system.

Another key to remember, you have the source code for the kernel, use it and learn from it. It's a great tool to understand the relationships between hardware and software. For example, think about the timer on the Integrator board. From a C program when the times() function is called, where does this time come from? It must be related to the hardware timer and the kernel must use the timer to provide the the answers about elapsed time.

When I started thinking about this article on debugging I had good intentions to use eclipse to index the kernel source and use the debug perspective in eclipse to connect to QEMU and debug. Over the last couple of years I have tried eclipse a few times and despite my best intentions I just can't enjoy using it. I find there are just too many settings hidden under Properties just to configure it to start the cross debugger, use the remote protocol, specify the file to load, and it seems to go on and on. Even though I didn't try all that long, if it takes more than 3 tries of tweaking settings it's not worth it. I can easily use command-line gdb or ddd and start debugging instantly. I think eclipse is actually popular so if you are interested in eclipse I found one article specific to QEMU and another general one about configuring eclipse for the Linux kernel.

Below is a screen shot of ddd debugging the Linux kernel for the ARM Integrator Virtual Platform.

As always, thanks for reading.