sxmo-image-builder: MAY PREVENT BOOTING - apk upgrade fails - boot partition too small

When running "apk upgrade" on a fresh sxmo 1.3.0 image, the following error is seen when trying to rebuild the initramfs:

(129/131) Upgrading sxmo-xdm-config-openrc (0.2.1-r1 -> 0.2.2-r0)
(130/131) Upgrading postmarketos-ui-sxmo (1.3.0-r1 -> 1.3.2-r0)
Executing postmarketos-ui-sxmo-1.3.2-r0.post-upgrade
 * rc-update: sxmo-pinephone already installed in runlevel `default'; skipping
 * rc-update: modemmanager already installed in runlevel `default'; skipping
 * rc-update: xdm already installed in runlevel `default'; skipping
(131/131) Purging llvm10-libs (10.0.1-r1)
Executing busybox-1.33.0-r4.trigger
Executing glib-2.66.7-r1.trigger
Executing eudev-3.2.10-r0.trigger
Executing dbus-1.12.20-r2.trigger
Executing ca-certificates-20191127-r5.trigger
Executing kmod-28-r0.trigger
depmod: WARNING: could not open modules.order at /lib/modules/5.10.12: No such file or directory
depmod: WARNING: could not open modules.builtin at /lib/modules/5.10.12: No such file or directory
Executing fontconfig-2.13.1-r3.trigger
Executing mkfontscale-1.2.1-r1.trigger
Executing postmarketos-mkinitfs-0.22-r0.trigger
==> initramfs: creating /boot/initramfs-postmarketos-allwinner
Scanning kernel module dependencies...
NOTE: ** modprobe warnings below can be ignored ** if your device does not run the
mainline kernel yet (most devices!) or if the related kernel options are enabled
with 'y' instead of 'm' (module).
 - deviceinfo: sun6i_mipi_dsi sun4i_drm pwm_sun4i sun8i_mixer anx7688 gpio_vibra
 - 00-default.modules: dm_crypt ext4 usb_f_rndis
==> kernel: device-tree blob operations
==> kernel: copying dtb allwinner/sun50i-a64-pinephone-1.1 allwinner/sun50i-a64-pinephone-1.2 to boot partition
==> initramfs: creating uInitrd
Image Name:   uInitrd
Created:      Wed Mar 17 23:04:31 2021
Image Type:   AArch64 Linux RAMDisk Image (uncompressed)
Data Size:    1442774 Bytes = 1408.96 KiB = 1.38 MiB
Load Address: 00000000
Entry Point:  00000000
==> kernel: creating uImage
Image Name:   postmarketos
Created:      Wed Mar 17 23:04:32 2021
Image Type:   AArch64 Linux Kernel Image (uncompressed)
Data Size:    19752968 Bytes = 19290.01 KiB = 18.84 MiB
Load Address: 80008000
Entry Point:  80008000
==> initramfs: creating /boot/initramfs-postmarketos-allwinner-extra
gzip: write error: No space left on device
cpio: write error: Broken pipe
Executing postmarketos-base-10-r0.trigger
Configuring a getty on port ttyS0 with baud rate 115200
Executing gtk-update-icon-cache-2.24.33-r0.trigger
OK: 792 MiB in 426 packages 

This is because:

  1. the initramfs-postmarketos-allwinner-extra file is being created on the /boot partition
  2. the new initramfs file is 46M in size, and there's only 33M free on /boot
  3. the new initramfs is 37% larger than the old initramfs(34M).
  4. libLLVM-10.so (62M) was replaced with libLLVM-11.so (89M).

Possible solutions:

  • Default boot_size be increased to at least 160M, with 200M preferable.
  • Create new initramfs files outside boot partition, then overwrite.
  • Trust users of pmbootstrap to predict and use safe value for boot_size option.
  • Have script remove existing initramfs from /boot before building new one

Additional information:

Comparison of libLLVM-##.so files sizes (output from du, in bytes):

62292864        old/usr/lib/libLLVM-10.so
88811280        new/usr/lib/libLLVM-11.so

df -h /boot output prior to upgrade:

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1  114M   75M   33M  81% /boot

df -h /boot output after (manual) upgrade:

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1  114M   87M   21M  81% /boot
Assigned to
3 years ago
3 years ago
bug high-priority

~tetrakist referenced this from #205 3 years ago

~trbl 3 years ago ยท edit

oh, so that killed my pp... it locked up during the update so I had to powercycle it, not seeing the output... have to re-flash... sigh...

~emulti 3 years ago*

confirmed- led to unbootable system for me and had to re-flash. Is it possible to use Jumpdrive to mount the partitions over usb network and resize them from a PC?

~emulti referenced this from #213 3 years ago

~dinkocar 3 years ago

I have just used Jumpdrive to resize the boot, and I am happy to report that Pinephone has happily booted after that operation :)

~proycon 3 years ago

Thanks for the detailed report. This is a very serious issue indeed. Everybody should be very careful upgrading!!!

I'll repeat ~tetrakist 's workaround (for existing upgrades) as mentioned on the chat:

sudo apk fix linux-postmarketos-allwinner

If it fails, "gzip: short write: No space left on device, do :

sudo rm /boot/initramfs-postmarketos-allwinner-extra
sudo apk fix linux-postmarketos-allwinner

~proycon 3 years ago

I added some instructions to our installation guide on how to move an existing postmarketos/sxmo system from SD card to the eMMC, allow enlarging the /boot partition in the process:


~ollieparanoid 3 years ago

Summary of what has been done in pmOS to address this (2021-03-21):

  • In postmarketos-mkinitfs, the new initramfs-extra is not generated next to the old one. Instead, it is generated in /tmp. This way it does not run out of space anymore, and prevents the issue here. (The old logic was supposed to do more-or-less an atomic replacement of the old file, more on that below.)
  • pmbootstrap has been patched to double the default boot partition size, 256 MiB instead of 128 MiB.
  • All new images, including the first images since v21.03 was announced, are now built with the bigger boot partition
  • With the bigger boot partition size, in theory it's now possible to do an atomic replacement of all files on the boot partition again. The plan is to update postmarketos-mkinitfs to check available size first, and if there is enough space, generate the new files next to the old one and replace them if they were built successfully. However, the postmarketos-mkinitfs code is pretty much the oldest code in postmarketOS and I'm not happy with the quality of the code anymore (doesn't use set -e for example). I'll probably just rewrite it and implement this safety feature (replacing the files from same partition) while at it. I've also considered using existing initramfs build scripts, but the postmarketOS mkinitfs is also able to repackage the initfs into boot.img and other formats needed to boot on android devices etc. So I'm leaning towards just rewriting the few LOC we already have. Details: https://gitlab.com/postmarketOS/pmaports/-/issues/1019

=> I think this can be closed.

~proycon REPORTED FIXED 3 years ago

Thanks for the update, that looks like a good solution!

Register here or Log in to comment, or comment via email.