MangoPI MQ Dual boot speed

Here we discuss getting the MangoPI MQ Dual to boot to a prompt in 1.5 seconds

The MangoPI MQ Dual is a tiny board based around the AllWinner T113-S3

This is an interesting processor because of its cost (less than $5 USD in medium quantities at time of writing), integrated memory (128MB), and relatively full Linux support.

Because of the 'small' nature of this CPU, it lends itself to more deeply embedded projects. However, when used in this manner the relatively 'heavy' nature of Linux can become an issue, as it has impacts on build sizes, boot time etc...

In the instructions below we're building a completely open source system from a combination of U-Boot, Linux kernel & Buildroot. This is all based on mainline upstream sources, which gives a huge amount of assurance about on-going support and the quality of development. The configuration of these pieces has been adjusted to minimise size (and functionality to some degree as a consequence) to demonstrate a plausible minimal Linux system as a starting point for building something. The focus is on boot speed.

Timing breakdown

The unit boots in the following rough phases/durations:

1. 250ms: Loading U-Boot from the SD card, initialising DDR, running U-Boot
2. 600ms: Loading Linux and root filesystem from the SD card and starting the Linux kernel
3. 400ms: Linux kernel initialising all drivers and mounting the initramfs root filesystem
4. 200ms: Userspace execution

The biggest aspects of this are really reading content from the SD card. Unfortunately the U-Boot driver for the SunXI MMC device inside the AllWinner T113-S3 does not give us great performance. We are currently seeing ~6MiB per second, on a card that should be capable of many times that. If further improvements were needed, this would be the area to optimise.

Boot speed tips & tricks

There isn't anything particularly special about the image that is constructed here - the fundamental C source code has not been change in any way. What we have done is tweak the configurations in such a way to remove features we don't want, and adjust some default values. Here are the main areas that have been adjusted:

Baudrates

Embedded Linux systems typically use a serial port as their primary console for development. This is normally not exposed on the final product. For various historic reasons, the default for this is set to 115200 baud. This is horribly slow. Modern systems have no problems with much higher baudrates on UARTs, some up to 12 Megabaud. For our purposes, 1.5 Megabaud is a good balance between speed, compatibility and standardisation.

We also add the quiet parameter to the Linux kernel bootargs to reduce its console output during boot. See ArchLinux wiki for more details. Full kernel logs are still available via dmesg if required

Configuration

U-Boot by default uses a 3 second boot delay to give you the opportunity to stop it at the prompt to run adhoc boot commands. While retaining the ability to do this is helpful, there is no need to have a delay at all. U-Boot will look for characters sent in the time between power on and the completion of the board initialisation. Any character sent then will interrupt the default boot if the boot delay value is set to 0. On the Mango Pi MQ this still gives us a few hundred milliseconds, which is sufficient.

This has been adjusted by setting CONFIG_BOOTDELAY=0 in the U-Boot configuration.

Storage access

This covers two things - don't hit the storage more frequently than you need, and don't store more than you need. To avoid hitting the storage too frequently we have dropped the U-Boot programmable environment. So you cannot make non-volatile changes to U-Boot without rebuilding U-Boot itself. This can actually be a benefit to reliability on embedded systems as any issues in the U-Boot environment can render the unit unbootable. So removing the ability to change it reduces some risk. It also means we don't have to look to storage to find a U-Boot environment file.

We have combined the kernel, device tree and filesystem into a single image which is loaded into memory in a continuous read.

We have reduced the functionality in the filesystem & kernel to the bare minimum, thus reducing image size (3.5MB at this stage).

Reduced functionality

We have removed as much as possible from the Linux kernel & Buildroot filesystem to bring the image size down.

In Buildroot we are using a statically built musl based busybox executable. This removes the need for libc.so, by integrating it with busybox. However if more binaries are added, this would get duplicated, eventually no longer being worth while.

In Linux we have turned off module support, and removed various subsystems that we're not using (wifi, sound etc...)

Other options

If the non-volatile root filesystem makes development awkward, moving over the using the EXT4 partition directly would be an option. This could also have some benefits to boot speed as only the needed files would be accessed during boot.

[0.000002 0.000002]
[0.000218 0.000216] U-Boot SPL 2024.10-rc1 (Sep 17 2024 - 11:21:52 +1200)
[0.000802 0.000584] DRAM: 128 MiB
[0.003019 0.002217] Trying to boot from MMC1
[0.199349 0.196330]
[0.199475 0.000126]
[0.199490 0.000015] U-Boot 2024.10-rc1 (Sep 17 2024 - 11:21:52 +1200) Allwinner Technology
[0.200530 0.001040]
[0.200546 0.000016] CPU:   Allwinner RModel: MangoPi MQ-R-T113
[0.201054 0.000508] DRAM:
[0.224077 0.023023] Core:  39 devices, 17 uclasses, devicetree: separate
[0.224673 0.000596] WDT:   Not starting watchdog@20500a0
[0.225035 0.000362] MMC:   mmc@4020000: 0, mmc@4021000: 1
[0.231221 0.006186] Loading Environment from nowhere... OK
[0.231650 0.000429] In:    serial@2500c00
[0.231819 0.000169] Out:   serial@2500c00
[0.231979 0.000160] Err:   serial@2500c00
[0.233635 0.001656] Net:   No ethernet found.
[0.235225 0.001590] Hit any key to stop autoboot:  0
[0.837554 0.602329] 3482343 bytes read in 578 ms (5.7 MiB/s)
[0.838717 0.001163] ## Loading kernel from FIT Image at 45000000 ...
[0.839962 0.001245]    Using 'config' configuration
[0.840907 0.000945]    Trying 'kernel' kernel subimage
[0.841317 0.000410]      Description:  Linux kernel
[0.841675 0.000358]      Type:         Kernel Image
[0.842026 0.000351]      Compression:  uncompressed
[0.842383 0.000357]      Data Start:   0x450000c8
[0.842728 0.000345]      Data Size:    2969640 Bytes = 2.8 MiB
[0.843210 0.000482]      Architecture: ARM
[0.843477 0.000267]      OS:           Linux
[0.843765 0.000288]      Load Address: 0x42000000
[0.844101 0.000336]      Entry Point:  0x42000000
[0.844451 0.000350]    Verifying Hash Integrity ... OK
[0.844864 0.000413] ## Loading ramdisk from FIT Image at 45000000 ...
[0.845434 0.000570]    Using 'config' configuration
[0.845811 0.000377]    Trying 'ramdisk' ramdisk subimage
[0.846217 0.000406]      Description:  buildroot
[0.846515 0.000298]      Type:         RAMDisk Image
[0.846792 0.000277]      Compression:  uncompressed
[0.847060 0.000268]      Data Start:   0x452d97a8
[0.847311 0.000251]      Data Size:    493041 Bytes = 481.5 KiB
[0.847678 0.000367]      Architecture: ARM
[0.847873 0.000195]      OS:           Linux
[0.848084 0.000211]      Load Address: unavailable
[0.848452 0.000368]      Entry Point:  unavailable
[0.848744 0.000292]    Verifying Hash Integrity ... OK
[0.849067 0.000323] ## Loading fdt from FIT Image at 45000000 ...
[0.849483 0.000416]    Using 'config' configuration
[0.849766 0.000283]    Trying 'fdt' fdt subimage
[0.850030 0.000264]      Description:  Flattened Device Tree blob
[0.850456 0.000426]      Type:         Flat Device Tree
[0.850762 0.000306]      Compression:  uncompressed
[0.851083 0.000321]      Data Start:   0x452d519c
[0.851346 0.000263]      Data Size:    17823 Bytes = 17.4 KiB
[0.851700 0.000354]      Architecture: ARM
[0.851899 0.000199]    Verifying Hash Integrity ... OK
[0.852197 0.000298]    Booting using the fdt blob at 0x452d519c
[0.852549 0.000352]    Loading Kernel Image to 42000000
[0.852805 0.000256]    Loading Ramdisk to 47cf2000, end 47d6a5f1 ... OK
[0.853170 0.000365]    Loading Device Tree to 47cea000, end 47cf159e ... OK
[0.860799 0.007629]
[0.860817 0.000018] Starting kernel ...
[0.860983 0.000166]
[1.188615 0.327632] [    0.001454] /cpus/cpu@0 missing clock-frequency property
[1.209622 0.021007] [    0.001513] /cpus/cpu@1 missing lock-frequency property
[1.364434 0.154812] Saving 256 bits of non-creditable seed for next boot
[1.369073 0.004639] Starting syslogd: OK
[1.379926 0.010853] Starting klogd: OK
[1.383684 0.003758] Running sysctl: OK
[1.398598 0.014914] Starting network: OK
[1.416731 0.018133] Starting crond: OK
[1.521718 0.104987]
[1.521754 0.000036] Welcome to Buildroot
[1.521923 0.000169] buildroot login: