There are three vital components that are required to successfully boot Linux on ARM:

• Kernel image: This should be self-evident :p
• Flattened Device Tree: A structured binary blob of data that the kernel parses to identify hardware elements.
• RootFS: A file system that can be mounted at / in order to load init. Here, we will use a ramdisk image.

Although these components can be manually loaded and configured for boot from U-Boot, it's preferable to package them together in a Flattened μImage Tree (FIT). Similarly to how bl1 and bl2 know how to load the binaries that comprise the FIP we generated in the previous lab, so does bl33 know how to boot the system from a FIT.

In the following tasks we will create each individual component and package them together. Finally, we will create a partition table (with one FAT partition) on the board's eMMC and store the FIT only to then load it and boot it from the bl33 shell.

Clone the kernel from the official GitHub repo, using the v6.6 tag. We recommend you to use this git cloning technique to avoid fetching the entire git history (requiring several GBs of disk space).

Since you've built U-Boot previously, you should be somewhat familiar with the Kbuild system. Start by generating the build configuration from its default configuration template. Optionally, run a menuconfig to inspect the Linux kernel options available.

In order to build the kernel image, simply make with the required variables (see the warning above!).

Note that the kernel image you will be including in the FIP is called Image. Unlike vmlinux which is an ELF file (i.e.: contains useless sections, including .debuginfo if you want to debug the kernel), Image is a boot executable image, made specifically for you to jump straight in and start executing. After the build process finalizes, find the Image file within the <linux-kernel-src>/arch/arm64/boot/ output directory.

While waiting for the build, explore the Linux source using your favorite code editor (especially the device trees)!

./scripts/clang-tools/gen_compile_commands.py

Luckily, the FDT for our platform is also included in Linux. It's name should be imx8mq-pico-pi.dtb (generated from its source counterpart, imx8mq-pico-pi.dts).

If for some reason it wasn't built alongside the kernel Image, check out the dtbs make target.

Usually there are two approaches to generating the root filesystem: bootstrapping from a source of pre-compiled binaries (e.g.: debootstrap, pacstrap, etc.) or building them yourself (e.g.: Yocto, Buildroot.) The former is usually preferred when working on desktop environments. The latter allows you to fine tune everything that is installed on your system. For example, the OpenSSH package available via pacstrap will come with a default build of sshd (the SSH server). On some critical infrastructure however, you might want to harden the SSH server binary by using a custom dynamic loader instead of ld-linux which allows LD_PRELOAD hooks.

Between Yocto and Buildroot, we generally opt for Buildroot. While Yocto has its advantages (e.g.: wide industry adoption, extensive device support, etc.), it also has a very steep learning curve and consumes significant amounts of resources (i.e.: ~50G storage space for a build). Although it has a layer-based build customization feature (think Dockerfiles extending base images), we argue that this makes it more difficult to comprehend the contents of what is being built.

Buildroot on the other hand is geared towards simplicity and ease of use. Being based on Kbuild and Makefile (same as Linux and U-Boot) makes it instantly familiar to most developers. And even if you are new to this, while Yocto requires you to read entire books in order to utilize it properly, Buildroot can be summed up in a 1k LoC Makefile.

Clone the official Buildroot repo.

Find an existing defconfig for our platform (the imx8mq) and take a look at what it contains. Notice how the following config options are enabled:

BR2_LINUX_KERNEL=y
BR2_TARGET_ARM_TRUSTED_FIRMWARE=y
BR2_TARGET_UBOOT=y

Yes, Buildroot also integrates the bootloaders into its build system. However, keeping the components separate makes it easier to appply patches and debug problems. What we want from Buildroot is a minimal root filesystem and nothing more. So, after generating .config, enter menuconfig and make a few changes:

• Disable all components that we've already integrated in our build environment (starting with the ones above).
• Use an external, custom, pre-installed toolchain (remember CROSS_COMPILE? check the menu for it!).
• Use systemd as the init system instead of BusyBox.
  • Make sure systemd-logind is included in the build, or you may have this problem.
• Choose bash as the default shell instead of sh.
• Enable password login for the root user, then set a password.
• Include a text editor PACKAGE (e.g.: vim or nano).
• Include the coreutils PACKAGE.
• Generate an uncompressed CPIO image from the output file system (see the Note below).

$ cd /path/to/rootfs
$ find . | cpio -o > /path/to/ramdisk.cpio

Feel free to add / remove anything else you want. Anything goes as long as you end up with a functioning system. After completing Step 3, maybe return and see if there's anything left to trim down to speed up the build process

Not much to it, really. Once everything's done, check out the output/ directory. What does it contain? Where is your CPIO archive?

Create a staging directory and copy everything that we've obtained from the previous three tasks. Then, create an Image Tree Source (ITS) file. We'll be referring to it as linux.its but the name doesn't really matter. What matters is the content:

/dts-v1/;

/ {
    description = "ASS - Linux FIT image for Pico Pi";
    #address-cells = <1>;

    images {
        kernel {
            description = "Linux kernel";
            data = /incbin/("Image");
            type = "kernel";
            arch = "arm64";
            os = "linux";
            compression = "none";
            load = <XXX>;
            entry = <XXX>;
        };
        fdt {
            description = "Device tree";
            data = /incbin/("imx8mq-pico-pi.dtb");
            type = "flat_dt";
            arch = "arm64";
            compression = "none";
            load = <YYY>;
        };
        initrd {
            description = "Ramdisk";
            data = /incbin/("rootfs.cpio");
            type = "ramdisk";
            arch = "arm64";
            os = "linux";
            compression = "none";
            load = <ZZZ>;
        };
    };

    configurations {
        default = "normal-boot";

        normal-boot {
            description = "Normal boot config";
            kernel = "kernel";
            fdt = "fdt";
            ramdisk = "initrd";
        };
    };
};

Let's unpack this:

• The file starts with /dts-v1/;, identifying it as a Device Tree Source.
• Next, we have a root / node with two child nodes: images and configurations.
  • images contains the description of each binary that will need to be loaded by U-Boot when processing the FIT.
    • Each image entry contains an incbin directive that tells mkimage to copy paste the contents of the specified file into the output DTB.
    • Each image also contains a load property, specifying the address where the data will be placed.
    • In addition to load, the kernel image also has an entry property specifying where U-Boot will jump when relinquishing control to the Linux kernel.
  • configurations contains sets of image configurations that can be chained.
    • The only configuration we have is also the default: normal-boot.
    • Notice how it has pre-defined attributes for kernel, fdt, ramdisk. These are not simple binary blobs; instead, they each have a role to play in the boot sequence. E.g., U-Boot will know to take the load address of the fdt image and pass it via a certain register (decided by convention) to the kernel, informing it where in memory to look for the FDT.

Note how we replaced the load and entry addresses with placeholders such as XXX. Replace these with whatever addresses you want such as the binaries do not overlap once U-Boot starts loading them.

Once all the addresses are filled in, generate the Image Tree Blob (ITB) using mkimage. Best not to use the imx-mkimage; instead, install mkimage using your distro's package manager.

$ mkimage -f linux.its linux.itb

There are many ways to make the FIT image available to U-Boot. For this task we chose a method that may seem roundabout, but can serve as a starting point toward making our board's configuration persistent across resets (i.e.: no longer using SDP each time we restart it.)

The goal is to add a FAT partition on the eMMC storage device and on that partition, place the FIT image. bl33 has the drivers necessary to parse the partition table and interact with the FAT32 file system.

Later in this exercise we will see how uuu can be used to overwrite the eMMC. Unfortunately, it will overwrite it indiscriminately starting at the very beginning. This means that an attempt to just flash the FIT image onto the eMMC will delete the partition table, rendering it useless to bl33. To circumvent this issue, we will create a disk image, containing the following:

• A partition table: needed in order to identify different partitions on the eMMC.
• ~10MB of empty space: this is where you can place the FIP image (i.e.: the one with the bootloaders and firmware) to be loaded automatically during an eMMC boot (refer to the Jumper configuration part of the previous lab).
• A FAT32 formatted partition containing the FIT image. Other file system types such as ext4 may be a bit too exotic for some bootloaders out there, but everyone should know how to interpret FAT.

Start of by creating an empty file. If your FIT image is ~70MB, then 100MB should suffice. No need to be conservative.

$ truncate --size 100M disk.img

Next, use fdisk to create a partition on disk.img.

Now that we have a partition, we want to format it with a FAT32 filesystem. Normally, the OS would make this painfully easy for us, by creating /dev/ entries for each partition. For example, if you plugged in a USB stick identified as /dev/sda with two partitions, the kernel would also generate the sda1 and sda2 entries for you to target with mkfs or to mount. As of now, the kernel does not recognize a random file that we just created as a storage device. So let's change that:

# -a : add new disk
# -v : verbose output helps identify the new device
$ partx -a -v disk.img 

The partx tool made our disk.img file known to the kernel and asked for it to be scanned for partitions. The kernel obliged and created a loopback device (i.e.: a file-backed storage device).

We've finally arrived at the point where we can format the partition, mount it and copy the fit image onto it:

# create FAT32 file system 
$ mkfs.fat -F 32 /dev/loop0p1
 
# mount file system (assuming /mnt is empty)
$ mount /dev/loop0p1 /mnt
 
# copy the FIT image
$ cp linux.itb /mnt
 
# unmount the file system
$ umount /mnt

# deregister our disk image as a storage device
$ partx -d /dev/loop0
 
# delete the loopback device that was allocated for us
$ losetup -D /dev/loop0

For this task we are going to follow the TechNexion guide for flashing the eMMC. Download their pre-compiled uuu binary since it comes with a few extra scripts and a custom bl2 downloads the disk image via SDP and uses it to overwrite the eMMC instead of booting into it, like our SPL.

We will use u-boot's Mass Storage Device emulation to present the board's eMMC flash memory as host device and write our image:

u-boot> ums mmc 0
# keep it running (i.e., do not ctrl+c)!

Simply use the dd tool or another image writer tool to burn the disk.bin image on the USB device presented by the board:

# first, run lsblk and determine the device's address:
lsblk
# look at the disk capacities and choose the 16G one
# PLEASE BE CAREFUL! TAKE NOTE TO IGNORE YOUR MAIN HOST PARTITIONS!
# OTHERWISE, YOU MAY LOSE PERSONAL DATA!
dd if=disk.bin of=/dev/<sdaX?> bs=4k

First things first: let's check if the flashing process was succesful. Reload your FIP using uuu and get back into the U-Boot shell.

u-boot=> # list MMC devices (or rather, the slots)
u-boot=> mmc list
FSL_SDHC: 0 (eMMC)
FSL_SDHC: 1

u-boot=> # select MMC device 0
u-boot=> mmc dev 0
switch to partitions #0, OK
mmc0(part 0) is current device

u-boot=> # show partitions on currently selected MMC device
u-boot=> mmc part

Partition Map for MMC device 0  --   Partition Type: DOS

Part    Start Sector    Num Sectors     UUID            Type
  1     20480           184320          5a0f642d-01     83
  
u-boot=> # show contents of / directory on MMC device 0, partition 1
u-boot=> # assuming that the partition in question is FAT
u-boot=> fatls mmc 0:1
 72709475   linux.itb

1 file(s), 0 dir(s)

In order for U-Boot to parse the FIT image and extract its components at their respective load addresses, we first need to bring the entire FIT image in main memory (use the load or fatload command for this). Choose a load address where the FIT image will not overlap with any of the three components when unpacked, or with the relocated bl33 code, e.g. >= 50000000.

This step is optional, but gives us the chance to see how U-Boot can parse a DTB file and extract its information. In the following example we assume that you've loaded the FIT image at address 0x 8000 0000 (i.e.: 2GB, 1GB offset in RAM):

u-boot=> # mark 0x 8000 0000 as the starting address of a device tree
u-boot=> fdt addr 0x80000000

u-boot=> # show contents of specific nodes in the RAM-based DTB
u-boot=> fdt list /
/ {
        timestamp = <0x64b55028>;
        description = "ASS - Linux FIT image for Pico Pi";
        #address-cells = <0x00000001>;
        images {
        };
        configurations {
        };
};

u-boot=> fdt list /configurations
configurations {
        default = "normal-boot";
        normal-boot {
        };
};

u-boot=> fdt list /configurations/normal-boot
normal-boot {
        description = "Normal boot config";
        kernel = "kernel";
        fdt = "fdt";
        ramdisk = "initrd";
};

Use the bootm <your_itb_addres> command, passing it the starting address of the FIT image.

And now things get interesting…

If you faithfully followed the instructions up to this point, you'll be glad to know that you are precisely 3 bugs away from having a working Linux-based system. So, let's get started:

After executing bootm, you may notice some successful loads of the FDT and ramdisk, but the following error during the kernel loading process:

uncompressed: uncompress error -28

Congratulations! The kernel is finally booting. I'm certain you're thankful for keeping that Makefile up to date, right? But what's this?

[    1.878803] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    1.887072] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.5.0-rc1-00248-gb6e6cc1f78c7 #1
[    1.894996] Hardware name: TechNexion PICO-PI-8M (DT)
[    1.900051] Call trace:
[    1.902501]  dump_backtrace+0x90/0xe8
[    1.906183]  show_stack+0x18/0x24
[    1.909510]  dump_stack_lvl+0x48/0x60
[    1.913182]  dump_stack+0x18/0x24
[    1.916503]  panic+0x31c/0x378
[    1.919567]  mount_root_generic+0x254/0x324
[    1.923762]  mount_root+0x16c/0x330

First time seeing a kernel panic may be a bit daunting, but the reason looks to be pretty clear:

Unable to mount root fs on unknown-block(0,0)

The kernel needs to know the backing device (and partition) for the root file system in order to mount it at /. Apparently, U-Boot did not know how to tell it to use the ramdisk as rootfs, but chances are that the kernel does know about it.

This is not documented in the command's help message but bootm relies on an environment argument called bootargs to pass the kernel its command line arguments. This should be a good place to start our investigation:

u-boot=> printenv bootargs
bootargs=console=ttymxc0,115200,115200 rdinit=/linuxrc clk_ignore_unused

From rdinit=/linuxrc we notice that the obsolete change_root procedure is being used. Since we don't actually have a persistent rootfs, let's just treat the ramdisk as root.

Now that the kernel panic has been solved, the only remaining issue is a freeze right after forking into User Space. The final kernel log message should be indicative of the underlying problem:

Welcome to Buildroot
buildroot login:
[    9.473402] random: crng init done
[   33.757425] buck1: disabling

Try to decompile the device tree that we include in the FIT image and that is passed to Linux. What is buck1?
Apply the patch mentioned in the tips section below, then recompile the DTB and the FIT image. This should solve the problem.