Hi,
I’m on a Rasp4b-8GB, with Freedombox 24.3 (stable / backports enabled). My FBX image is from the freedombox.org website.
I’ve noticed since Debian12, my Raspi has not been removing old kernels from /root/firmware. With only 250MB of space, the partition gets filled on every two kernel upgrades. If I don’t manually intervene, this prevents FBX auto updating the kernel.
Has anyone else had this kind of issue? If so, any pointers for help would be appreciated.
Thanks.
Any logs that indicate it tried and possible errors?
Hi @timmy
I’m looking, but cant seem to find any… Trying to pipe journalct to grep with various kernel versions but nothing.
I thin I dont know how to use journalctl to get me what I want. Any suggestions for me to follow?
Thanks.
Quick search suggests logs for apt (the underlaying package manager) are in
/var/log/apt/term.log
nothing out of the ordinary…
i have a hunch this has more to do with raspi than with apt
raspi does not boot with grub, it has its own firmware folder to boot from. so, although apt works as it should with \boot, it doesnt seem to configure anything related to \boot\firmware (raspi specific)
before debian 12, raspi had an application called raspi-firmware. it had its issues and was depreciated with bookworm…
I’ve not handled FBX on raspi so I’m ignorant and unable help.
thanks @timmy, appreciate your time
Hi @Ged296123,
In FreedomBox, removal of old kernels is handled by unattended-upgrades. It has has mechanism to detect kernel packages (with APT::VersionedKernelPackages in /etc/apt/apt.conf.d/01autoremove). It uses default settings to remove unused dependencies and unused kernel packages (Unattended-Upgrade::Remove-Unused-Kernel-Packages “true”; ). It will not remove the latest kernel and the currently running kernel package. All of this can be debugged by running: unattended-upgrades --debug --dry-run. Logs are also present in /var/log/unattended-upgrades/. You should message such as Removing unused kernel packages and Keeping the following auto-removable package....
The second part of the equation is that /boot/firmware is specific to raspberry pi. Installing the kernel package (and removing it) does not add files (or remove) to the /boot/firmware. Instead the script /etc/kernel/postinst.d/z50-raspi-firmware script from raspi-firmware package takes care of this duty. It also removes any kernels that are not present in /boot/ directory from /boot/firmware directory.
Could you please run the following commands and post their output:
uname -adpkg -l linux-*unattended-upgrades --debug --dry-run.dpkg -l raspi-firmwarels -l /boot and ls -l /boot/firmware.Hi @sunil
Thanks for the help.
Outputs of requested commands below:
The currently running kernel version:
uname -a
Linux freedombox 6.1.0-18-arm64 #1 SMP Debian 6.1.76-1 (2024-02-01) aarch64 GNU/Linux
Show the list of currently installed kernel packages: dpkg -l linux-*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================================-============-============-===============================================
ii linux-base 4.9 all Linux image base package
un linux-doc-6.1 <none> <none> (no description available)
ii linux-image-6.1.0-18-arm64 6.1.76-1 arm64 Linux 6.1 for 64-bit ARMv8 machines (signed)
un linux-image-6.1.0-18-arm64-unsigned <none> <none> (no description available)
ii linux-image-arm64 6.1.76-1 arm64 Linux for 64-bit ARMv8 machines (meta-package)
un linux-image-generic <none> <none> (no description available)
un linux-initramfs-tool <none> <none> (no description available)
un linux-latest-modules-6.1.0-18-arm64 <none> <none> (no description available)
ii linux-libc-dev:arm64 6.1.76-1 arm64 Linux support headers for userspace development
un linux-perf <none> <none> (no description available)
See if unattened-upgrades is refusing to remove kernel packages: unattended-upgrades --debug --dry-run
Starting unattended upgrades script
Allowed origins are: origin=Debian,codename=bookworm,label=Debian, origin=Debian,codename=bookworm,label=Debian-Security, origin=Debian,codename=bookworm-security,label=Debian-Security, o=Debian Backports,n=bookworm-backports,l=Debian Backports
Initial blacklist:
Initial whitelist (not strict):
Marking not allowed <apt_pkg.PackageFile object: filename:'/var/lib/apt/lists/deb.debian.org_debian_dists_bookworm-updates_main_i18n_Translation-en' a=stable-updates,c=main,v=12-updates,o=Debian,l=Debian arch='' site='deb.debian.org' IndexType='Debian Translation Index' Size=83585 ID:5> with -32768 pin
Marking not allowed <apt_pkg.PackageFile object: filename:'/var/lib/apt/lists/deb.debian.org_debian_dists_bookworm-updates_main_binary-arm64_Packages' a=stable-updates,c=main,v=12-updates,o=Debian,l=Debian arch='arm64' site='deb.debian.org' IndexType='Debian Package Index' Size=66088 ID:4> with -32768 pin
Applying pinning: PkgFilePin(id=5, priority=-32768)
Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/var/lib/apt/lists/deb.debian.org_debian_dists_bookworm-updates_main_i18n_Translation-en' a=stable-updates,c=main,v=12-updates,o=Debian,l=Debian arch='' site='deb.debian.org' IndexType='Debian Translation Index' Size=83585 ID:5>
Applying pinning: PkgFilePin(id=4, priority=-32768)
Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/var/lib/apt/lists/deb.debian.org_debian_dists_bookworm-updates_main_binary-arm64_Packages' a=stable-updates,c=main,v=12-updates,o=Debian,l=Debian arch='arm64' site='deb.debian.org' IndexType='Debian Package Index' Size=66088 ID:4>
Using (^linux-.*-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^kfreebsd-.*-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^gnumach-.*-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^.*-modules-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^.*-kernel-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^linux-.*-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^kfreebsd-.*-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^gnumach-.*-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^.*-modules-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^.*-kernel-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$) regexp to find kernel packages
Using (^linux-.*-6\.1\.0\-18\-arm64$|^linux-.*-6\.1\.0\-18$|^kfreebsd-.*-6\.1\.0\-18\-arm64$|^kfreebsd-.*-6\.1\.0\-18$|^gnumach-.*-6\.1\.0\-18\-arm64$|^gnumach-.*-6\.1\.0\-18$|^.*-modules-6\.1\.0\-18\-arm64$|^.*-modules-6\.1\.0\-18$|^.*-kernel-6\.1\.0\-18\-arm64$|^.*-kernel-6\.1\.0\-18$|^linux-.*-6\.1\.0\-18\-arm64$|^linux-.*-6\.1\.0\-18$|^kfreebsd-.*-6\.1\.0\-18\-arm64$|^kfreebsd-.*-6\.1\.0\-18$|^gnumach-.*-6\.1\.0\-18\-arm64$|^gnumach-.*-6\.1\.0\-18$|^.*-modules-6\.1\.0\-18\-arm64$|^.*-modules-6\.1\.0\-18$|^.*-kernel-6\.1\.0\-18\-arm64$|^.*-kernel-6\.1\.0\-18$) regexp to find running kernel packages
pkgs that look like they should be upgraded:
Fetched 0 B in 0s (0 B/s)
fetch.run() result: 0
Packages blacklist due to conffile prompts: []
No packages found that can be upgraded unattended and no pending auto-removals
The list of kept packages can't be calculated in dry-run mode.
Version of raspi-firmware package:
dpkg -l raspi-firmware
dpkg-query: no packages found matching raspi-firmware
please note that i believe raspi-firmware was removed from the debian installer after v.12.1. I believe this post may shed more light into the matter.
Contents of /boot and /boot/firmware:
ls -l /bootandls -l /boot/firmware
me@freedombox:~$ ls -l /boot
total 68284
-rw-r--r-- 1 root root 291002 Şub 1 11:05 config-6.1.0-18-arm64
drwxr-xr-x 2 root root 16384 Oca 1 1970 firmware
-rw-r--r-- 1 root root 36981739 Şub 11 11:28 initrd.img-6.1.0-18-arm64
-rw-r--r-- 1 root root 83 Şub 1 11:05 System.map-6.1.0-18-arm64
-rw-r--r-- 1 root root 32622528 Şub 1 11:05 vmlinuz-6.1.0-18-arm64
me@freedombox:~$ ls -l /boot/firmware
total 89880
-rwxr-xr-x 1 root root 27406 Eki 18 16:31 bcm2711-rpi-400.dtb
-rwxr-xr-x 1 root root 27434 Eki 18 16:31 bcm2711-rpi-4-b.dtb
-rwxr-xr-x 1 root root 27323 Eki 18 16:31 bcm2711-rpi-cm4-io.dtb
-rwxr-xr-x 1 root root 14816 Eki 18 16:31 bcm2837-rpi-3-a-plus.dtb
-rwxr-xr-x 1 root root 14993 Eki 18 16:31 bcm2837-rpi-3-b.dtb
-rwxr-xr-x 1 root root 15349 Eki 18 16:31 bcm2837-rpi-3-b-plus.dtb
-rwxr-xr-x 1 root root 14355 Eki 18 16:31 bcm2837-rpi-cm3-io3.dtb
-rwxr-xr-x 1 root root 14667 Eki 18 16:31 bcm2837-rpi-zero-2-w.dtb
-rwxr-xr-x 1 root root 52460 Haz 15 2023 bootcode.bin
-rwxr-xr-x 1 root root 125 Ağu 12 2023 cmdline.txt
-rwxr-xr-x 1 root root 577 Şub 11 11:37 config.txt
-rwxr-xr-x 1 root root 3170 Haz 15 2023 fixup4cd.dat
-rwxr-xr-x 1 root root 5399 Haz 15 2023 fixup4.dat
-rwxr-xr-x 1 root root 8379 Haz 15 2023 fixup4db.dat
-rwxr-xr-x 1 root root 8379 Haz 15 2023 fixup4x.dat
-rwxr-xr-x 1 root root 3170 Haz 15 2023 fixup_cd.dat
-rwxr-xr-x 1 root root 7262 Haz 15 2023 fixup.dat
-rwxr-xr-x 1 root root 10228 Haz 15 2023 fixup_db.dat
-rwxr-xr-x 1 root root 10226 Haz 15 2023 fixup_x.dat
-rwxr-xr-x 1 root root 36981739 Şub 11 11:37 initrd.img-6.1.0-18-arm64
-rwxr-xr-x 1 root root 803964 Haz 15 2023 start4cd.elf
-rwxr-xr-x 1 root root 3744808 Haz 15 2023 start4db.elf
-rwxr-xr-x 1 root root 2249280 Haz 15 2023 start4.elf
-rwxr-xr-x 1 root root 2996680 Haz 15 2023 start4x.elf
-rwxr-xr-x 1 root root 803964 Haz 15 2023 start_cd.elf
-rwxr-xr-x 1 root root 4816712 Haz 15 2023 start_db.elf
-rwxr-xr-x 1 root root 2973536 Haz 15 2023 start.elf
-rwxr-xr-x 1 root root 3720360 Haz 15 2023 start_x.elf
-rwxr-xr-x 1 root root 32622528 Şub 11 11:37 vmlinuz-6.1.0-18-arm64
kindly note that upon noticing that apt was installing the recent kernel and the old weren’t being removed i manually intervened with the following:
a. removed old kernels manually with following (x refers to old kernels)
sudo apt remove linux-image-x-arm64
sudo rm /boot/firmware/vmlinuz-x-arm64
sudo rm /boot/firmware/initrd.img-x-arm64
b. copied recent kernel files to /boot/firmware (y refers most recent kernel version)
sudo cp /boot/initrd.img-y-arm64 /boot/firmware/
sudo cp /boot vmlinuz-y-arm64 /boot/firmware/
c. manually modified the /boot/firmware/config.txt file to point to the recent kernel at boot.
# Do not modify this file!
#
# It is automatically generated upon install or update of either the
# firmware or the Linux kernel.
#
# If you need to set boot-time parameters, do so via the
# /etc/default/raspi-firmware, /etc/default/raspi-firmware-custom or
# /etc/default/raspi-extra-cmdline files.
# Switch the CPU from ARMv7 into ARMv8 (aarch64) mode
arm_64bit=1
enable_uart=1
upstream_kernel=1
kernel=vmlinuz-y-arm64
# For details on the initramfs directive, see
# https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=10532
initramfs initrd.img-y-arm64
yes I know - it does say do not modify : ) i was out of options at the time…
Again, thank you for the time and support.
Ged
*edited: was trying to solve issue at the time, cant be sure if raspi-firware is installed default.
The raspi-firmware package missing on your installation is the root cause of the problem as far as I can tell. It is meant to do keep the latest firmware in /boot/firmware, copy the latest kernel into /boot/firmware, and update the config.txt file. I wonder why this package is missing. The forum discussion and the two Debian bugs that it lists relate to not having the raspi-firmware package in amd64 architecture, which I don’t think is relevant for us (arm64 architecture for raspberry pi). Installing the raspi-firmware package should take of solving these issues for you.
Also, unattended-upgrades should have taken care of obsolete removing linux-image-* packages. And you should not have had to remove them manually. This may have happened because the latest kernel was never put into /boot/firmware/. Latest kernel was never booted into. So, unattended upgrades refused to remove the old, currently running kernel. This fix in this case too is to install raspi-firmware package.
FreedomBox images should come with raspi-firmware package preinstalled. And since your image is stable, no major change in packages could have remove it. Did you go through distribution upgrades process to Bookworm?
Anyone else using the Raspberry Pi hardware face issue with missing raspi-firmware package?
Hi @sunil
ok. ill proceed with installing raspi-firmware and observe how the pi behaves in future kernel upgrades.
thanks for the effort and your time. ill mark this posted as solved for now, will revert here to continue searching if the problem persists.
cheers,
ged