In my previous post I talked about how to compile the required components to be able to build a bootable system from scratch. In this post I’ll talk about how to package what we had build earlier.
First we need a media which can be plugged in to QEMU through which it can boot our system. The following way can be used to do this,
- Create a loopback device and format a file system onto it. We’ll try to keep the boot partition and the work partition as separate but you can keep them same if you like.
- Install BusyBox binaries on the formatted media.
- Install Linux kernel and it’s drivers on formatted media.
Using a Loop Device
We can create a 2GiB image easily using loop device. We’ll need to do a couple of things though
- Create partitions on that disk (our loop device).
- Format it and generate UUID. We’ll see it’s use later.
#Create a loop device
#We write in multiple of 512 bytes which is sector size.
dd if=/dev/zero of=2G_img bs=1 seek=2GiB count=1024 Creating Partitions
Parted is a nice utility which can be used on files as well as block devices. It also allows to create GPT based partitions which is really nice. In this tutorial we’ll use parted in interactive mode, however you can use –script option to automate the whole process.
~>sudo parted 2G_img
(parted) mklabel gpt
(parted) mkpart primary ext4 2MiB 264MiB
(parted) mkpart primary ext4 265MiB 98%
(parted) toggle 1 boot
(parted) print
Model: (file)
Disk /home/pranay/rootfs_images/2G_img: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 2097kB 277MB 275MB ext4 primary boot, esp
2 278MB 1053MB 775MB ext4 primary
In the above commands we make two partitions
- Our boot partition that’ll house our kernel, drivers and initramfs if any.
- Second one is the partition where all work files would be stored, some of which would be editable.
NOTE: We intentionally start out at 2MiB instead of 0 since GPT headers also require some space so we leave that part. Same is true for MBR partition type.
NOTE: We didn’t eat up all the remaining disk since GPT backup copy is stored at the end of disk. So we left that part out as well.
Installing Files
The partitions we created on our fake disk, a file essentially, can be mounted using kpartx . The following command shows how to do this,
> sudo kpartx -a 2G_img
> ls /dev/mapper/
control loop0p1 loop0p2
#/dev/mapper/loop0p1 is partition 1 etc...
#mount partition 1 in /mnt
sudo mount /dev/mapper/loop0p1 /mnt Formatting Partitions
Before we can install files we need to create a file system on our partition. Since we already. marked partition type to be ext4 in GPT table, let’s format our partition using mkfs.ext4.
> sudo mkfs.ext4 /dev/mapper/loop0p1
mke2fs 1.46.4 (18-Aug-2021)
Discarding device blocks: done
Creating filesystem with 268288 1k blocks and 67056 inodes
Filesystem UUID: b17475f2-6ca3-407a-bbc2-7d17646c7e4f
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
> sudo mkfs.ext4 /dev/mapper/loop0p2
mke2fs 1.46.4 (18-Aug-2021)
Discarding device blocks: done
Creating filesystem with 189184 4k blocks and 47328 inodes
Filesystem UUID: 4e1e4861-9fc7-4af2-9500-42f6d810e8e4
Superblock backups stored on blocks:
32768, 98304, 163840
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
Installing Kernel and Drivers
First move in the directory where you build kernel. Then the following command would install drivers and kernel to correct location.
NOTE: Install drivers first and then kernel.
>sudo mount /dev/mapper/loop0p1 /mnt
#move to kernel build dir
kernel_src>sudo INSTALL_MOD_PATH=/mnt make modules_install
# %<--Snipped Output >%
kernel_src> sudo mkdir /mnt/boot
kernel_src> sudo INSTALL_PATH=/mnt/boot ARCH=arm64 \
CROSS_COMPILE=aarch64-none-linux-gnu- make install
sh ./arch/arm64/boot/install.sh 5.16.0-rc1pks-badass-109181-g42eb8fdac2fc \
arch/arm64/boot/Image System.map "/mnt/boot"
Installing normal kernel At this point we don’t need to do any further modifications in our partition 1. Thus let’s now prepare partition 2 and get the things we need in there.
Preparing Work Partition
First mount the Partition 2 and install BusyBox in it. You can do this using the following command
busybox> sudo --preserve-env=PATH ARCH=arm64 \
CROSS_COMPILE=aarch64-none-linux-gnu- make \
CONFIG_PREFIX=/mnt install Copying the libraries
The above is not enough to be able to run BusyBox applets. Since we did a dynamic linking we would require that standard libraries are present in the required path. For this we need to copy the libraries provided in the cross compilation toolchain to our system.
NOTE: We’ll need the libraries from non-linux toolchain only.
#The following is the location of libc libraries
#of ARM cross toolchain on my system.
>cd Downloads/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/aarch64-none-linux-gnu/libc
#Copy everything into our work partition.
>sudo rsync -avrP * /mnt
Additional directories
We would need some additional directories, viz, proc, dev and sys for mounting procfs and sysfs in our system. These would be populated by init, however busybox’s init (linuxrc) doesn’t do that. So we’ll have to mount this when we drop into shell.
#Directory Structure before creating essential dirs
> ls /mnt/
bin etc lib lib64 linuxrc lost+found sbin usr var
#Directory Structure After creating essential dirs
ls /mnt/
bin dev etc lib lib64 linuxrc lost+found proc sbin sys usr var
Time to boot
At this time we need to unmount our work partition and we need to boot it with QEMU.
NOTE: The kernel supports cortex-a57 for this software virtualisation hence we use that cpu model for QEMU virtual machine as well.
qemu-system-aarch64 -M virt -m 1G -cpu cortex-a57 \
-kernel u-boot -nographic -drive if=none, \
file=2G_img,format=raw,id=hd0 -device virtio-blk-pci \
,drive=hd0 It doesn’t boot
Well yes and no, it boots u-boot however we need to tell u-boot where certain things are, namely
- Device Tree
- Location of kernel
- root device for kernel to mount
- Init program to run after it mounts root device
For this particular machine the device tree is created on the fly and is available at $fdt_addr. We also need to load kernel from disk into memory so that u-boot can pass control to it. Thus steps 1 -4 can be accomplished in a sequence of steps as shown below.
NOTE: A device tree is a compiled information about the devices present on the machine(or board) and their respective addresses and ranges.
=> load virtio 0:1 ${kernel_addr_r} /boot/vmlinux-5.16.0-rc1pks-badass-109181-g42eb8fdac2fc
=> booti ${kernel_addr_r} - ${fdt_addr}
## Flattened Device Tree blob at 40000000
Booting using the fdt blob at 0x40000000
Loading Device Tree to 000000007ed08000, end 000000007ee0afff ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 5.16.0-rc1pks-badass-109181-g42eb8fdac2fc (pranay@linux-fqjg) (aarch64-none-elf-gcc (GNU Toolchain for the A-profile Architecture 10.3-2021.07 (arm-10.29)) 10.3.1 20210621, GNU ld (GNU Toolchain for the A-profile Architecture 10.3-2021.07 (arm-10.29)) 2.36.1.20210621) #1 SMP PREEMPT Fri Nov 19 18:35:21 IST 2021
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] efi: UEFI not found.
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x7fdf1b80-0x7fdf3fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
%<------SNIPPED--------->%
[ 0.612463] CPU features: 0x2,200019c2,00000846
[ 0.612808] Memory Limit: none
[ 0.613294] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
As you can see we were in the correct direction however, the kernel was not able to figure out root device and hence couldn’t mount it. Since it couldn’t mount root device there was no init to be found and it panicked.
So we now need to tell u-boot to pass the root device to our kernel, if you look closely in the traces you’ll see the following lines
[ 0.449637] virtio_blk virtio1: [vda] 2097154 512-byte logical blocks (1.07 GB/1.00 GiB)
[ 0.474787] vda: vda1 vda2 These two vda partitions correspond to our two partitions. Since we know the second partition contains linuxrc we’ll try to pass vda2 and linuxrc as boot arguments to kernel as shown below,
=> load virtio 0:1 ${kernel_addr_r} /boot/vmlinux-5.16.0-rc1pks-badass-109181-g42eb8fdac2fc
=> kernel_args="root=/dev/vda2 init=/linuxrc"
=> setenv bootargs ${kernel_args}
=> booti ${kernel_addr_r} - ${fdt_addr}
## Flattened Device Tree blob at 40000000
Booting using the fdt blob at 0x40000000
Loading Device Tree to 000000007ed08000, end 000000007ee0afff ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 5.16.0-rc1pks-badass-109181-g42eb8fdac2fc (pranay@linux-fqjg) (aarch64-none-elf-gcc (GNU Toolchain for the A-profile Architecture 10.3-2021.07 (arm-10.29)) 10.3.1 20210621, GNU ld (GNU Toolchain for the A-profile Architecture 10.3-2021.07 (arm-10.29)) 2.36.1.20210621) #1 SMP PREEMPT Fri Nov 19 18:35:21 IST 2021
[ 0.000000] Machine model: linux,dummy-virt
%<-----SNIP------->%
[ 0.839659] Run /linuxrc as init process
can't run '/etc/init.d/rcS': No such file or directory
Please press Enter to activate this console.
/ #
If we do a mount command here we won’t have any output since it’s not mounted by busy box’s linuxrc (init) . Thus we must do it ourselves and as well as sysfs. Also since rootfs is mounted read only we’ll also have to remount it in rw mode.
/ # mount -t proc none /proc
/ # mount
/dev/root on / type ext4 (ro,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=487096k,nr_inodes=121774,mode=755)
none on /proc type proc (rw,relatime)
/ # mount -o remount,rw /
/ # mount
/dev/root on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=487096k,nr_inodes=121774,mode=755)
none on /proc type proc (rw,relatime)
Will vda2 always be the correct disk?
Not always, but in this case you can assume it. However if we wanted to automate this image creation and booting then we can’t rely on device names. Luckily there’s another way out, PARTUUID.
There are two of these
- UUID is a file system specific Identifier and requires additional support.
- PARTUUID is partition’s UUID and has support from kernel.
While trying to automate this image building we can run blkid on our partition and get the PARTUUID. For example on my machine the partitions I created have the following information from blkid.
> sudo blkid /dev/mapper/loop0p2
/dev/mapper/loop0p2: UUID="4e1e4861-9fc7-4af2-9500-42f6d810e8e4" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="5cea0bf1-66b9-45dd-82a6-5f7e8c830c8b"
>sudo blkid /dev/mapper/loop0p1
/dev/mapper/loop0p1: UUID="b17475f2-6ca3-407a-bbc2-7d17646c7e4f" BLOCK_SIZE="1024" TYPE="ext4" PARTLABEL="primary" PARTUUID="772b47da-f139-4092-b795-df65de3894ea" Ok but I still have to type this in u-boot
Not really, if you examine the output in u-boot and check in code you’ll find that there’s a script which u-boot looks for when trying to boot. This script’s name by default is boot.scr and this is a compiled script not a text file.
We can place all our commands in that script and since we’ll also know PARTUUID we can put that in there as well. The following shows the contents of uboot.txt used for creating boot.scr. (or boot.scr.img).
You can read about this in the file distro_bootcmd.h in u-boot code.
echo "Loading kernel from $devtype - device $devnum partition 1"
load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/vmlinux-5.16.0-rc1pks-badass-109181-g42eb8fdac2fc
kernel_args="root=PARTUUID=612cb556-7a7d-46a6-8ac6-ccd2ca96401f init=/linuxrc"
echo "Setting kernel arguments as $kernel_args"
setenv bootargs ${kernel_args}
booti ${kernel_addr_r} - ${fdt_addr}
u-boot/tools/mkimage -T \
script -O linux -d uboot.txt boot.scr We can now put this in our boot partition, i.e our first partition in the boot directory. And when we run QEMU again it’ll find this boot.scr file and execute the commands inside it.
> qemu-system-aarch64 -M virt -m 1G -cpu cortex-a57 -kernel \
u-boot -nographic -drive if=none,\
file=2G_img,format=raw,id=hd0 -device virtio-blk-pci,\
drive=hd0
U-Boot 2021.10-dirty (Dec 05 2021 - 17:29:22 +0530)
DRAM: 1 GiB
Flash: 64 MiB
Loading Environment from Flash... *** Warning - bad CRC, using default environment
In: pl011@9000000
Out: pl011@9000000
Err: pl011@9000000
Net: eth0: virtio-net#32
Hit any key to stop autoboot: 0
starting USB...
No working controllers found
USB is stopped. Please issue 'usb start' first.
scanning bus for devices...
Device 0: unknown device
Device 0: 1af4 VirtIO Block Device
Type: Hard Disk
Capacity: 1024.0 MB = 1.0 GB (2097154 x 512)
... is now current device
Scanning virtio 0:1...
Found U-Boot script /boot/boot.scr
434 bytes read in 1 ms (423.8 KiB/s)
## Executing script at 40200000
Loading kernel from virtio - device 0 partition 1
21502464 bytes read in 7 ms (2.9 GiB/s)
Setting kernel arguments as root=PARTUUID=5cea0bf1-66b9-45dd-82a6-5f7e8c830c8b init=/linuxrc
## Flattened Device Tree blob at 40000000
Booting using the fdt blob at 0x40000000
Loading Device Tree to 000000007ed06000, end 000000007ee08fff ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 5.16.0-rc1pks-badass-109181-g42eb8fdac2fc (pranay@linux-fqjg) (aarch64-none-elf-gcc (GNU Toolchain for the A-profile Architecture 10.3-2021.07 (arm-10.29)) 10.3.1 20210621, GNU ld (GNU Toolchain for the A-profile Architecture 10.3-2021.07 (arm-10.29)) 2.36.1.20210621) #1 SMP PREEMPT Fri Nov 19 18:35:21 IST 2021
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] efi: UEFI not found.
[ 0.000000] NUMA
%<------SNIPPED-------->%
[ 0.625947] devtmpfs: mounted
[ 0.657413] Freeing unused kernel memory: 2176K
[ 0.658422] Run /linuxrc as init process
can't run '/etc/init.d/rcS': No such file or directory
Please press Enter to activate this console.
/ #