Ubuntu on ARM is Growing

Based on the title, you probably guessed I'm talking about growing in popularity. Yes, Ubuntu on ARM is growing in popularity, but here I'm referring to growing in size.

About a month ago I talked to a very sharp engineer from Canonical, the makers of Ubuntu, named Loic Miner. During the discussion of Ubuntu on ARM he asked me to look at a bug report in the Ubuntu bug tracking system related to the Lucid release. This is version 10.04 and is scheduled to be released next week. The problem was related to Lucid for the ARM Versatile platform not working on qemu. The messages indicated the kernel was having trouble finding the root filesystem.

[ 1.866499] Trying to unpack rootfs image as initramfs...
[ 1.875076] rootfs image is not initramfs (junk in compressed archive); looks like an initrd
[...]
[ 5.347930] RAMDISK: Couldn't find valid RAM disk image starting at 0.

In Exploring the Virtual Platform Part 4 I described how qemu takes a kernel and filesystem as arguments and just starts running. This is very convenient (when it works) since it bypasses the use of a full boot loader.  In many embedded systems a boot loader such as Das U-Boot is used to initialize the hardware and prepare for the kernel to start. Hopefully I'll cover u-boot in a future article.

Digging into this specific bug was a good review of how qemu starts the kernel and how it puts the filesystem into memory so the kernel can find it.

In the qemu source tree the file arm_boot.c is where all of this takes place.
The key addresses are given as defines.

#define KERNEL_ARGS_ADDR 0x100
#define KERNEL_LOAD_ADDR 0x00010000
#define INITRD_LOAD_ADDR 0x00800000
   
These three addresses define where the arguments for the kernel are placed in memory, where the kernel is loaded into memory, and where the filesystem given by the -initrd argument is placed in memory.

The other key part of the process is the "bootloader". As it says in the comment, it's pretty small.

/* The worlds second smallest bootloader.  Set r0-r2, then jump to kernel.  */
static uint32_t bootloader[] = {
  0xe3a00000, /* mov     r0, #0 */
  0xe3a01000, /* mov     r1, #0x?? */
  0xe3811c00, /* orr     r1, r1, #0x??00 */
  0xe59f2000, /* ldr     r2, [pc, #0] */
  0xe59ff000, /* ldr     pc, [pc, #0] */
  0, /* Address of kernel args.  Set by integratorcp_init.  */
  0  /* Kernel entry point.  Set by integratorcp_init.  */
};

Some of the parts of the bootloader code are filled in dynamically when the kernel is loaded by the function arm_load_kernel()

 bootloader[1] |= info->board_id & 0xff;
 bootloader[2] |= (info->board_id >> 8) & 0xff;
 bootloader[5] = info->loader_start + KERNEL_ARGS_ADDR;
 bootloader[6] = entry;

You can map the array values that are modified to the definition above and see that the board id information is put into r1 and the address of the kernel arguments and the kernel entry point are filled in.

I found a page about Booting ARM Linux to be the most valuable description of how the process works.

Here is the overview of the steps:

• Configure the memory system.
• Load the kernel image at the correct memory address.
• Optionally load an initial RAM disk at the correct memory address.
• Initialise the boot parameters to pass to the kernel.
• Obtain the ARM Linux machine type
• Enter the kernel with the appropriate register values.

The requirements to start the kernel:

• The CPU must be in SVC (supervisor) mode with both IRQ and FIQ interrupts disabled.
• The MMU must be off, i.e. code running from physical RAM with no translated addressing.
• Data cache must be off
• Instruction cache may be either on or off
• CPU register 0 must be 0
• CPU register 1 must be the ARM Linux machine type
• CPU register 2 must be the physical address of the parameter list

The other interesting code in the qemu file arm_boot.c related to the setup of the kernel parameters.  The qemu function set_kernel_args() maps very closely to the description of
/* ATAG_MEM */
/* ATAG_INITRD2 */
/* ATAG_CMDLINE */
/* ATAG_BOARD */
/* ATAG_END */

provided on the Booting ARM Linux page. There is a series of memory writes that setup the kernel arguments in the memory area defined by KERNEL_ARGS_ADDR.

When I looked at why the Ubuntu ARM kernel would not boot on the ARM Versatile model I was stumped at first. Everything in qemu seemed fine. The size of the kernel should fit in the kernel load area. Because the kernel is loaded first and then the filesystem is loaded into memory I thought that if there was any corruption the filesystem would appear OK because it was loaded last.

It was a bit of a guess, but I suspected the kernel was too big for the memory area of size (INITRD_LOAD_ADDR - KERNEL_LOAD_ADDR) but after realizing that the compressed vmlinuz must also be uncompressed into this area it was clear the memory was too small. It's interesting to note that most embedded Linux systems use the compressed image since it's faster to read a smaller file from the storage media and uncompress it in memory than reading a bigger file and skipping the uncompress.

Sure enough, I changed the initrd address to something bigger and the OS booted fine.
-#define INITRD_LOAD_ADDR 0x00800000
+#define INITRD_LOAD_ADDR 0x00a00000

So in the end the issue was caused by the growth of Ubuntu!

I was interested to see how the Ubuntu development process worked. Upon reporting the fix to Loic, he quickly had a change checked in. It's amazing to me that all of the software that makes up a Linux distribution like Ubuntu can be maintained. I read there are over 40,000 packages available for Ubuntu. It seems very little of this code is original and exists somewhere else "upstream", just like qemu. So the bug fix is made in the Ubuntu version of qemu and somehow this has to propagate back to the qemu repository (or maybe not). If it doesn't then the next version of qemu that is imported into Ubuntu will have the same bug unless somebody merges this fix. Then there is all the coordination between the Ubuntu bug tracking and the linking of the same bugs in multiple bug tracking systems. Although there seems like a lot of duplication in an uncoordinated way, it's really amazing to see how all of this software is managed. Best of all the entire process is available for anybody to see and to participate in.

OVP also offers a model that does the exact same thing as the qemu code I described today. The model is called SmartLoaderArmLinux and also takes a kernel and initrd filesystem and just boots. I wonder where this idea came from?

I'm looking for any ideas readers have about how to identify this bug in an easier way. qemu has a connection to gdb as it's debugging interface; maybe this could have been used to catch the corruption. There is also gdb on the qemu code itself (which I did spend some time doing). Commercial Virtual Platform tools have more advanced debugging features that may be of more help. Please share your ideas.

Jason Andrews