Update - when to panic?

I am able to access tt-rss via my phone app but the VPN is like 80% down. Plinth is also down with the following error for like 1 day right now. A restart didn’t resolve it. Should I just wait? I don’t have ssh access.

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.


Apache/2.4.56 (Debian) Server at freedombox.local Port 443

I was able to reproduce a failed upgrade on a Raspberry Pi 4. The FreedomBox web interface became unavailable, being stuck in the same error loop described above by @Avron. Here’s what you can do if you have SSH or console access:
0. (if you use SSH) install screen, to stay connected even if the SSH session disconnects:
sudo apt install -y screen

  1. screen [enter, then press space]
  2. sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  3. after the upgrade finished, I ran apt update again, and saw that the freedombox package was held, so ran: sudo apt-mark unhold freedombox
  4. Then I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again.
  5. Now my /boot partition didn’t have enough free space to update initramfs. If you are facing the same issue, you can free up some space and then update initramfs. Please only do this if update-initramfs is failing for you:
    5.1. sudo mkdir /root/kernel-backup
    5.2. sudo mv /boot/firmware/initrd.img-5* /root/kernel-backup/
    5.3. sudo mv /boot/firmware/vmlinuz-5* /root/kernel-backup/
    5.4 DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  6. sudo reboot
1 Like

I am at step (2) but now it is asking me about config files, and I do not want to make the wrong choice and break my setup.

Any suggestions for what to do when it asks:

Configuration file X
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.

Note that I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade and did not expect to be asked about config files.

Edited to add: I see I already asked about the “noninteractive” option here: Unattended-upgrades vs. DEBIAN_FRONTEND=noninteractive apt-get upgrade vs. apt-get upgrade -y

From the above link re the “noninteractive” prompt, I see this answer from @jvalleroy:

unattended-upgrades will not upgrade packages that have a conffile prompt. This is a good thing, because we often release new versions of FreedomBox to properly handle conffile prompts. FreedomBox will check for packages to be upgraded on a regular basis.

Does this mean I am already in a bad situation because I am upgrading a package with a conffile prompt? Several people in my family are now using the box and I really do not want to have to do a fresh install and potentially lose their data and have to set everything up all over again.

Any help would be greatly appreciated!

Update: In the end only one config file had a conffile prompt: /etc/janus/janus.jcfg.

After searching the internet I decided to chose Y to install the package maintainer’s version. However, I first made a backup by logging in to a new ssh session and copying the old config file.

I also had the issue that my /boot partition didn’t have enough free space to update initramfs. However, I solved this by running sudo apt autoremove, which seemed to free up enough space to proceed. I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again. Everything went well.

I then continued with steps (3)-(4) from Update - when to panic? - #17 by nbenedek.

I did NOT need to run step (5).

After rebooting I can log in via SSH. I still do not have access to the Plinth interface, but maybe it will need a few minutes to boot everything.

1 Like

Hi @dgj ,

I did all things above AND changed the firewalld-settings mentioned in https://discuss.freedombox.org/t/debian-12-bookworm-release-and-upgrading/2591/7.

It worked on two boxes.

Cheers Uli

1 Like

Yes, changing to “DefaultZone=external” in /etc/firewalld solved it.

I can access Plinth and ran Diagnostics. Everything has passed!

Thanks, everyone :slightly_smiling_face:

P.s. Nice to see some familiar names popping up in the forum again.

2 Likes

Yes, changing to “DefaultZone=external” in /etc/firewalld solved it.

I can access Plinth and ran Diagnostics. Everything has passed!

The change in /etc/firewalld/firewalld.conf of DefaultZone=external to DefaultZone=public seemed suspicious since, according to my notes, I had never touched that file. Choosing to accept the packager maintainer’s version when I attempted the manual upgrade (selecting Y at the conffile prompt) seems to have been the wrong choice for me. That said, I noticed that one of my FreedomBoxes that did successfully automatically upgrade without any issue (besides a minor Mediawiki hiccup) ended up with DefaultZone=external in its firewalld.conf file.

The box that had a successful automatic upgrade from Bullseye to Bookworm (originally spun up 2022-09-01) now has 3 files in /etc/firewalld/:

  • firewalld.conf with DefaultZone=external. Modified 2023-06-11.
  • firewalld.conf.old with DefaultZone=public. Modified 2023-01-06.
  • firewalld.conf.dpkg-old with DefaultZone=external. Modified 2022-08-17.

The FreedomBox that had the failed upgrade to from Bullseye to Bookworm (originally spun up on 2022-05-09) and which I reverted to an evening backup had in its backup image:

  • firewalld.conf with DefaultZone=external. Modified 2022-05-09.
  • firewalld.conf.old with DefaultZone=public. Modified 2021-02-01.

When I next attempt an upgrade, I’ll try making sure DefaultZone=external remains in /etc/firewalld/firewalld.conf.

donotd

2 Likes

I have now been without my Freedombox for four days. I feel I have tried most of the above (I am confused about this Firewall stuff - is that the original problem).

I just want it to work like it did on Saturday… been following the step by steps (I appreciate the suggestions) but I am getting this:

**matthew@freedombox**:**~**$ sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

Calculating upgrade... Done

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

1 not fully installed or removed.

After this operation, 0 B of additional disk space will be used.

Do you want to continue? [Y/n] y

Setting up initramfs-tools (0.142) ...

update-initramfs: deferring update (trigger activated)

Processing triggers for initramfs-tools (0.142) ...

update-initramfs: Generating /boot/initrd.img-6.1.0-9-armmp-lpae

W: Possible missing firmware /lib/firmware/imx/sdma/sdma-imx7d.bin for built-in driver imx_sdma

W: Possible missing firmware /lib/firmware/imx/sdma/sdma-imx6q.bin for built-in driver imx_sdma

zstd: error 70 : Write error : cannot write block : No space left on device

E: mkinitramfs failure zstd -q -9 -T0 70

update-initramfs: failed for /boot/initrd.img-6.1.0-9-armmp-lpae with 1.

dpkg: error processing package initramfs-tools (--configure):

installed initramfs-tools package post-installation script subprocess returned error exit status 1

Errors were encountered while processing:

initramfs-tools

E: Sub-process /usr/bin/dpkg returned an error code (1)

Any idea what I should do next?

My current thought is get rid of the Freedombox and go back to a hosted Matrix server or use Telegram and stop trying to be all ethical and open source - but hopefully that passes.

Hello, I’m no expert, but I believe this line might indicate that your boot partition might not have enough space to complete the operation.

Maybe you could try to run sudo apt autoremove and see if removes some of the old kernels. I believe you can target specific packages with sudo apt autoremove --purge <kernel-package-name>, but personally I would just try the previous command.

As I said, I’m no expert. Follow this advice at your own risk. I’d recommend that you do your due diligence before attempting this.

After freeing space in your boot partition, run the upgrade process again.

1 Like

You are right, I had the same issue (and mentioned it). I had 4 kernels and initrd files in /boot so I ran “sudo apt remove” for the two oldest ones. @matthewguy you can check the package names by running e.g. dpkg -l “linux*”

After this, you can manually launch the upgrade again (sudo apt update, sudo apt upgrade, sudo apt full-upgrade). After it is finished, you may still have the firewall issue (default zone set to public, need to change it to external). Hopefully everything will work then.

I managed to get the bookworm update installed (refreshing app list didn’t work due to an expired repo key), later the boot partition was full which I cleared. No my freedombox is at v23.6.2. The forums says 23.11 is the most recent. Is that version still not in stable and thus is it correct that that version is not yet installed?

I also have 23.6.2 after migrating to bookworm.

The post about 23.11 mentions experimental. According to https://packages.debian.org, 23.11 is still only in experimental (not even in unstable = sid). https://wiki.debian.org/DebianExperimental says that “experimental isn’t a complete distribution, it can work only as an extension of unstable.”

So to me it looks normal that we don’t have this version now.

1 Like

Update - I am back with a working Pioneer FreedomBox.

Thank you for your help everyone. I deleted the files causing my boot disk to be full, ran the update - all looked good and then it would not boot. I may have fiddled around too much and broke something in my attempts to make it all work. It was a busy week at work and I did not have the time (or patience) to sit down and be more forensic in my efforts.

In the end I got a blank microSD card and started again from scratch. It seemed the simpler option at that point.

I think ultimately it was the firewall issue that was messing with me but did not really understand that at the time. It has rather dented my faith in using the FB as anything but a chat server for the family.

I will try to ensure I do not end up in this position again when I have time (backups etc) but I am a little disappointed at the update process this time.

3 Likes

And another question, is the firewalld.conf with DefaultZone=public necessary to be changed back to DefaultZone=external ? I use wireguard vpn to access the VPN from the outside my LAN. Without VPN, the freedombox should not be reachable from outside the LAN. Everything seems to be working fine however.

My only issue on my raspberry pi for the upgrade was with Apache not starting due to this error:

Cannot load /usr/lib/apache2/modules/libphp7.4.so into server

Solved this problem by running:

sudo a2dismod php7.4
sudo systemctl restart apache2

All is well with Apache now and I can get to the website interface.

Unfortunately, this also applies to me. My sister had just asked how I could get her setup with a similar home server a couple months ago, and I am more hesitant now than I was before to recommend a FreedomBox.

That said, I love this project, I love self-hosting my data, and I would be interested in focusing on how to improve the upgrade experience for the future. Some open questions: What led to such a problematic upgrade? What needs to be done so it never happens again? How can we the community help?

2 Likes

I am not sure upgrade problems can be avoided in general.

A way to mitigate upgrade problems could be:

  • by default freedombox are not migrated automatically to a new stable Debian version
  • the migration is only attempted by users with some experience and willingness to deal with problems
  • these users report problems, try finding solutions, document them and write sufficient (and reasonably easy) guidance to fix all the detected problems
  • when that is ready, users are invited to check the guidance and upgrade

Some coordination may be useful, this forum is already a nice tool to share problems and solutions. If we want to do more than, that requires volunteers and and some kind of organization.

I am not sure how long would be needed until upgrade by everyone, perhaps one or two months?

3 Likes

Hello,

I find myself in full agreement with the idea you’ve put forward to mitigate upgrade problems in future Debian releases. The strategy you suggested - to initially limit the migration process to experienced users who are prepared to troubleshoot issues - seems like a very sensible approach.

Perhaps, FreedomBox could include a toggle option for ‘Advanced Users’ within the interface. Selecting this option would allow the distribution upgrade to proceed as normal for those users. Those who do not select the option would have their upgrade process delayed until all known issues have been identified and resolved. Users could be notified about the pending distribution upgrade and assured that it will be performed automatically in the near future once it’s safe to proceed.

As a platform built on Debian, FreedomBox has a reputation for stability, which is one of the main reasons users like us have trusted it to host and manage our sensitive information at home. Encountering issues during the upgrade process can understandably be disheartening, but it’s essential to remember that the upgrade issues are not a reflection of FreedomBox’s long-term performance or stability.

To share a bit of my own experience, I ended up performing the upgrade manually and was back up and running rather quickly. However, I did experience around five days of downtime while waiting for the unattended upgrade to do its work, which unfortunately didn’t transpire. This interruption significantly impacted my work and home organization routines, highlighting how integral my FreedomBox has become in my day-to-day activities.

But I’d like to encourage everyone who’s currently feeling frustrated - keep going. Yes, these upgrade issues are inconvenient, but remember, they are a part of a major update that comes every two years. The platform continues to improve, and it’s worth staying the course.

Looking forward to hearing more thoughts on this.

3 Likes