Update - when to panic?

I’ll try to reproduce the issue, I’ll get back with my findings.

I ran

$ sudo su -
Password: <enter user password here>
# dpkg --configure -a
# apt update
# apt -f install
# unattended-upgrade --debug
# apt install freedombox
# apt update
# apt full-upgrade

After reboot, trying to reach the web interface gives an error that the server is not secure.

I have the following problem with apache2:

root@fbox:/etc/apache2# journalctl --follow --unit=apache2.service
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:warn] [pid 1113:tid 1113] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox systemd[1]: Started apache2.service - The Apache HTTP Server.
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:warn] [pid 1167:tid 1167] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox apache-error[1168]: [http2:warn] [pid 1167:tid 1167] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
Jun 11 21:58:17 fbox apache-error[1168]: [mpm_prefork:notice] [pid 1167:tid 1167] AH00163: Apache/2.4.57 (Debian) mod_auth_pubtkt/0.13 OpenSSL/3.0.9 configured -- resuming normal operations
Jun 11 21:58:17 fbox apache-error[1168]: [core:notice] [pid 1167:tid 1167] AH00094: Command line: '/usr/sbin/apache2'

I replaced my domain name with “mydomain.tld”. To reach my freedombox, I only type “https://mydomain.eu” not “https://fbox.mydomain.eu”. I don’t know whether this has anything to do with the issue (it used to work before upgrade).

EDIT: I did the following:

Apparently, ejabberd and dovecot are running but they are unreachable :frowning:

Broke my Pioneer box.

My AMD btrfs server is up for previous services, but
WireGuard is broken.

SearX was removed as an installed app.

  • S

I am able to access tt-rss via my phone app but the VPN is like 80% down. Plinth is also down with the following error for like 1 day right now. A restart didn’t resolve it. Should I just wait? I don’t have ssh access.

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.


Apache/2.4.56 (Debian) Server at freedombox.local Port 443

I was able to reproduce a failed upgrade on a Raspberry Pi 4. The FreedomBox web interface became unavailable, being stuck in the same error loop described above by @Avron. Here’s what you can do if you have SSH or console access:
0. (if you use SSH) install screen, to stay connected even if the SSH session disconnects:
sudo apt install -y screen

  1. screen [enter, then press space]
  2. sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  3. after the upgrade finished, I ran apt update again, and saw that the freedombox package was held, so ran: sudo apt-mark unhold freedombox
  4. Then I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again.
  5. Now my /boot partition didn’t have enough free space to update initramfs. If you are facing the same issue, you can free up some space and then update initramfs. Please only do this if update-initramfs is failing for you:
    5.1. sudo mkdir /root/kernel-backup
    5.2. sudo mv /boot/firmware/initrd.img-5* /root/kernel-backup/
    5.3. sudo mv /boot/firmware/vmlinuz-5* /root/kernel-backup/
    5.4 DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  6. sudo reboot
1 Like

I am at step (2) but now it is asking me about config files, and I do not want to make the wrong choice and break my setup.

Any suggestions for what to do when it asks:

Configuration file X
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.

Note that I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade and did not expect to be asked about config files.

Edited to add: I see I already asked about the “noninteractive” option here: Unattended-upgrades vs. DEBIAN_FRONTEND=noninteractive apt-get upgrade vs. apt-get upgrade -y

From the above link re the “noninteractive” prompt, I see this answer from @jvalleroy:

unattended-upgrades will not upgrade packages that have a conffile prompt. This is a good thing, because we often release new versions of FreedomBox to properly handle conffile prompts. FreedomBox will check for packages to be upgraded on a regular basis.

Does this mean I am already in a bad situation because I am upgrading a package with a conffile prompt? Several people in my family are now using the box and I really do not want to have to do a fresh install and potentially lose their data and have to set everything up all over again.

Any help would be greatly appreciated!

Update: In the end only one config file had a conffile prompt: /etc/janus/janus.jcfg.

After searching the internet I decided to chose Y to install the package maintainer’s version. However, I first made a backup by logging in to a new ssh session and copying the old config file.

I also had the issue that my /boot partition didn’t have enough free space to update initramfs. However, I solved this by running sudo apt autoremove, which seemed to free up enough space to proceed. I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again. Everything went well.

I then continued with steps (3)-(4) from Update - when to panic? - #17 by nbenedek.

I did NOT need to run step (5).

After rebooting I can log in via SSH. I still do not have access to the Plinth interface, but maybe it will need a few minutes to boot everything.

1 Like

Hi @dgj ,

I did all things above AND changed the firewalld-settings mentioned in https://discuss.freedombox.org/t/debian-12-bookworm-release-and-upgrading/2591/7.

It worked on two boxes.

Cheers Uli

1 Like

Yes, changing to “DefaultZone=external” in /etc/firewalld solved it.

I can access Plinth and ran Diagnostics. Everything has passed!

Thanks, everyone :slightly_smiling_face:

P.s. Nice to see some familiar names popping up in the forum again.

2 Likes

Yes, changing to “DefaultZone=external” in /etc/firewalld solved it.

I can access Plinth and ran Diagnostics. Everything has passed!

The change in /etc/firewalld/firewalld.conf of DefaultZone=external to DefaultZone=public seemed suspicious since, according to my notes, I had never touched that file. Choosing to accept the packager maintainer’s version when I attempted the manual upgrade (selecting Y at the conffile prompt) seems to have been the wrong choice for me. That said, I noticed that one of my FreedomBoxes that did successfully automatically upgrade without any issue (besides a minor Mediawiki hiccup) ended up with DefaultZone=external in its firewalld.conf file.

The box that had a successful automatic upgrade from Bullseye to Bookworm (originally spun up 2022-09-01) now has 3 files in /etc/firewalld/:

  • firewalld.conf with DefaultZone=external. Modified 2023-06-11.
  • firewalld.conf.old with DefaultZone=public. Modified 2023-01-06.
  • firewalld.conf.dpkg-old with DefaultZone=external. Modified 2022-08-17.

The FreedomBox that had the failed upgrade to from Bullseye to Bookworm (originally spun up on 2022-05-09) and which I reverted to an evening backup had in its backup image:

  • firewalld.conf with DefaultZone=external. Modified 2022-05-09.
  • firewalld.conf.old with DefaultZone=public. Modified 2021-02-01.

When I next attempt an upgrade, I’ll try making sure DefaultZone=external remains in /etc/firewalld/firewalld.conf.

donotd

2 Likes

I have now been without my Freedombox for four days. I feel I have tried most of the above (I am confused about this Firewall stuff - is that the original problem).

I just want it to work like it did on Saturday… been following the step by steps (I appreciate the suggestions) but I am getting this:

**matthew@freedombox**:**~**$ sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

Calculating upgrade... Done

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

1 not fully installed or removed.

After this operation, 0 B of additional disk space will be used.

Do you want to continue? [Y/n] y

Setting up initramfs-tools (0.142) ...

update-initramfs: deferring update (trigger activated)

Processing triggers for initramfs-tools (0.142) ...

update-initramfs: Generating /boot/initrd.img-6.1.0-9-armmp-lpae

W: Possible missing firmware /lib/firmware/imx/sdma/sdma-imx7d.bin for built-in driver imx_sdma

W: Possible missing firmware /lib/firmware/imx/sdma/sdma-imx6q.bin for built-in driver imx_sdma

zstd: error 70 : Write error : cannot write block : No space left on device

E: mkinitramfs failure zstd -q -9 -T0 70

update-initramfs: failed for /boot/initrd.img-6.1.0-9-armmp-lpae with 1.

dpkg: error processing package initramfs-tools (--configure):

installed initramfs-tools package post-installation script subprocess returned error exit status 1

Errors were encountered while processing:

initramfs-tools

E: Sub-process /usr/bin/dpkg returned an error code (1)

Any idea what I should do next?

My current thought is get rid of the Freedombox and go back to a hosted Matrix server or use Telegram and stop trying to be all ethical and open source - but hopefully that passes.

Hello, I’m no expert, but I believe this line might indicate that your boot partition might not have enough space to complete the operation.

Maybe you could try to run sudo apt autoremove and see if removes some of the old kernels. I believe you can target specific packages with sudo apt autoremove --purge <kernel-package-name>, but personally I would just try the previous command.

As I said, I’m no expert. Follow this advice at your own risk. I’d recommend that you do your due diligence before attempting this.

After freeing space in your boot partition, run the upgrade process again.

1 Like

You are right, I had the same issue (and mentioned it). I had 4 kernels and initrd files in /boot so I ran “sudo apt remove” for the two oldest ones. @matthewguy you can check the package names by running e.g. dpkg -l “linux*”

After this, you can manually launch the upgrade again (sudo apt update, sudo apt upgrade, sudo apt full-upgrade). After it is finished, you may still have the firewall issue (default zone set to public, need to change it to external). Hopefully everything will work then.

I managed to get the bookworm update installed (refreshing app list didn’t work due to an expired repo key), later the boot partition was full which I cleared. No my freedombox is at v23.6.2. The forums says 23.11 is the most recent. Is that version still not in stable and thus is it correct that that version is not yet installed?

I also have 23.6.2 after migrating to bookworm.

The post about 23.11 mentions experimental. According to https://packages.debian.org, 23.11 is still only in experimental (not even in unstable = sid). https://wiki.debian.org/DebianExperimental says that “experimental isn’t a complete distribution, it can work only as an extension of unstable.”

So to me it looks normal that we don’t have this version now.

1 Like

Update - I am back with a working Pioneer FreedomBox.

Thank you for your help everyone. I deleted the files causing my boot disk to be full, ran the update - all looked good and then it would not boot. I may have fiddled around too much and broke something in my attempts to make it all work. It was a busy week at work and I did not have the time (or patience) to sit down and be more forensic in my efforts.

In the end I got a blank microSD card and started again from scratch. It seemed the simpler option at that point.

I think ultimately it was the firewall issue that was messing with me but did not really understand that at the time. It has rather dented my faith in using the FB as anything but a chat server for the family.

I will try to ensure I do not end up in this position again when I have time (backups etc) but I am a little disappointed at the update process this time.

3 Likes

And another question, is the firewalld.conf with DefaultZone=public necessary to be changed back to DefaultZone=external ? I use wireguard vpn to access the VPN from the outside my LAN. Without VPN, the freedombox should not be reachable from outside the LAN. Everything seems to be working fine however.

My only issue on my raspberry pi for the upgrade was with Apache not starting due to this error:

Cannot load /usr/lib/apache2/modules/libphp7.4.so into server

Solved this problem by running:

sudo a2dismod php7.4
sudo systemctl restart apache2

All is well with Apache now and I can get to the website interface.