Update - when to panic?

Support
6

If you have SSH access, you can issue the following commands to get some info about the update process: sudo journalctl --follow --unit=plinth.service and sudo tail -f /var/log/*

7

On my side, this command gives:

juin 11 18:24:37 fbox systemd[1]: plinth.service: Consumed 11.456s CPU time.
juin 11 18:24:43 fbox systemd[1]: plinth.service: Scheduled restart job, restart counter is at 48.
juin 11 18:24:43 fbox systemd[1]: Stopped plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:43 fbox systemd[1]: plinth.service: Consumed 11.456s CPU time.
juin 11 18:24:43 fbox systemd[1]: Started plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:54 fbox systemd[1]: plinth.service: Main process exited, code=exited, status=1/FAILURE
juin 11 18:24:54 fbox systemd[1]: plinth.service: Failed with result ‘exit-code’.
juin 11 18:24:54 fbox systemd[1]: plinth.service: Consumed 11.373s CPU time.
juin 11 18:24:55 fbox systemd[1]: Stopped plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:55 fbox systemd[1]: plinth.service: Consumed 11.373s CPU time.

I ran “systemctl stop plinth” (same for ejabberd) since that thing is repeating itself and brings nothing.

8

I have the same 503 error, just posting to confirm there is an issue.

I used a spare Micro Sd Card and a fresh installation, started the update via terminal, now going to wait a bit and see what happens.

9

When I run “apt update”, I have:

Hit:1 Index of /debian stable InRelease
Get:2 Index of /debian stable-updates InRelease [52.1 kB]
Get:3 Index of /debian bullseye-backports InRelease [49.0 kB]
Get:4 Index of /debian-security stable-security InRelease [48.0 kB]
Get:5 Index of /debian-security stable-security/main Sources [9184 B]
Get:6 Index of /debian-security stable-security/main armhf Packages [17.2 kB]
Get:7 Index of /debian-security stable-security/main Translation-en [7460 B]

Is it ok to have bullseye-backports when stable is now bookworm? (that comes from /etc/apt/sources.list.d/freedombox2.list). Should I remove that file before upgrading?

10

I found out that /boot is full and this is why update-initramfs fails.

/boot seems to include 4 initrd.img and 4 vmlinuz. (5.10.0-10/22/23 and 6.1.0.0-0). “uname -a” says that 6.1.0-0 is used so perhaps I can just remove 5.10.0-10/22 to make space?

11

I thought I’d do the same, but ended up with a server failing to serve any web pages (including the Plinth webUI) despite the apache2 service running. I had to restore from an evening snapshot of the server (an AWS Lightsail snapshot, not the FreedomBox backup).

During the manual upgrade ($ sudo apt update && sudo apt upgrade) I had to answer several prompts involving LDAP-related prompts (nslcd, libnss-ldapd) for which I just hit ‘Enter’, accepting the defaults, and hoped for the best. Some of these changes involved:

  • Configuring nslcd
    • LDAP sever URI : I accepted the default ldapi:/// .
  • Configuring nslcd
    • LDAP server search base: I accepted the default dc=thisbox .
  • Configuring popularity-contest
    • Participate in the package usage survey?: I accepted the default <No>.
  • Configuring libnss-ldapd
    • Name services to configure:: I kept the default choices selected:
      • [*] passwd
      • [*] group
      • [*] shadow
      • Note: Various error messages following the failed upgrade appear in journalctl logs mention nslcd and passwd which weren’t present in such logs prior to the upgrade attempt. I don’t know what to make of them.

I did get prompts for accepting or rejecting changes to configuration files I’d modified:

  • /etc/ssh/sshd_config. I accepted the package maintainer version. Some changes I noticed were:
    • PasswordAuthentication no: to disable password authentication, a change I remember making.
    • StreamLocalBindUnlink yes: to enable GnuPG Agent Forwarding; not essential to login via SSH; the change was one I made for the convenience of signing git commits with my local OpenPGP smartcard.
    • TrustedUserCAKeys /etc/ssh/lightsail_instance_ca.pub: Not my modification; a change I believe was already present in the Debian image I booted the AWS Lightsail instance from before installing the freedombox package back in 2022-05.
  • /etc/firewalld/firewalld.conf: I don’t believe I had ever touched this, so I just accepted the package maintainer version which contained, among other changes, a removal of AllowZoneDrifting=no and change of DefaultZone=external to DefaultZone=public.

At some point, failing to see the package freedombox get upgraded due to a hold, I decided to risk running $ sudo apt-mark unhold freedombox && sudo apt upgrade -y; I only did this because I had a recent backup of the entire server; the result was the Plinth webUI failing to start, and no apache2 pages being served (despite the # systemctl status apache2.service showing the service was running) despite being able to login via SSH.

After about half an hour of not seeing any significant CPU work, I decided to restore from an evening snapshot taken by Lightsail prior to the update; after starting the snapshot in a new machine instance, I unchecked the “Enable auto-update to next stable release” option in the System > Software Update section; now I’m waiting to see dust settle here on this forum before I make another upgrade attempt.

1 Like
12

I tried running sudo journalctl --follow --unit=plinth.service and I am getting the same error message.

Unfortunately I do not have a recent backup and thus will wait if there is any help for upgrading safely and without needing to answer any prompts.

Help much appreciated. My family is now regularly using the FreedomBox and they are feeling the downtime.

1 Like
13

I’ll try to reproduce the issue, I’ll get back with my findings.

14

I ran

$ sudo su -
Password: <enter user password here>
# dpkg --configure -a
# apt update
# apt -f install
# unattended-upgrade --debug
# apt install freedombox
# apt update
# apt full-upgrade

After reboot, trying to reach the web interface gives an error that the server is not secure.

I have the following problem with apache2:

root@fbox:/etc/apache2# journalctl --follow --unit=apache2.service
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:warn] [pid 1113:tid 1113] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox systemd[1]: Started apache2.service - The Apache HTTP Server.
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:warn] [pid 1167:tid 1167] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox apache-error[1168]: [http2:warn] [pid 1167:tid 1167] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
Jun 11 21:58:17 fbox apache-error[1168]: [mpm_prefork:notice] [pid 1167:tid 1167] AH00163: Apache/2.4.57 (Debian) mod_auth_pubtkt/0.13 OpenSSL/3.0.9 configured -- resuming normal operations
Jun 11 21:58:17 fbox apache-error[1168]: [core:notice] [pid 1167:tid 1167] AH00094: Command line: '/usr/sbin/apache2'

I replaced my domain name with “mydomain.tld”. To reach my freedombox, I only type “https://mydomain.eu” not “https://fbox.mydomain.eu”. I don’t know whether this has anything to do with the issue (it used to work before upgrade).

EDIT: I did the following:

Apparently, ejabberd and dovecot are running but they are unreachable :frowning:

15

Broke my Pioneer box.

My AMD btrfs server is up for previous services, but
WireGuard is broken.

SearX was removed as an installed app.

  • S
16

I am able to access tt-rss via my phone app but the VPN is like 80% down. Plinth is also down with the following error for like 1 day right now. A restart didn’t resolve it. Should I just wait? I don’t have ssh access.

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.4.56 (Debian) Server at freedombox.local Port 443

17

I was able to reproduce a failed upgrade on a Raspberry Pi 4. The FreedomBox web interface became unavailable, being stuck in the same error loop described above by @Avron. Here’s what you can do if you have SSH or console access:
0. (if you use SSH) install screen, to stay connected even if the SSH session disconnects:
sudo apt install -y screen

  1. screen [enter, then press space]
  2. sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  3. after the upgrade finished, I ran apt update again, and saw that the freedombox package was held, so ran: sudo apt-mark unhold freedombox
  4. Then I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again.
  5. Now my /boot partition didn’t have enough free space to update initramfs. If you are facing the same issue, you can free up some space and then update initramfs. Please only do this if update-initramfs is failing for you:
    5.1. sudo mkdir /root/kernel-backup
    5.2. sudo mv /boot/firmware/initrd.img-5* /root/kernel-backup/
    5.3. sudo mv /boot/firmware/vmlinuz-5* /root/kernel-backup/
    5.4 DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  6. sudo reboot
1 Like
18

I am at step (2) but now it is asking me about config files, and I do not want to make the wrong choice and break my setup.

Any suggestions for what to do when it asks:

Configuration file X
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.

Note that I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade and did not expect to be asked about config files.

Edited to add: I see I already asked about the “noninteractive” option here: Unattended-upgrades vs. DEBIAN_FRONTEND=noninteractive apt-get upgrade vs. apt-get upgrade -y

19

From the above link re the “noninteractive” prompt, I see this answer from @jvalleroy:

unattended-upgrades will not upgrade packages that have a conffile prompt. This is a good thing, because we often release new versions of FreedomBox to properly handle conffile prompts. FreedomBox will check for packages to be upgraded on a regular basis.

Does this mean I am already in a bad situation because I am upgrading a package with a conffile prompt? Several people in my family are now using the box and I really do not want to have to do a fresh install and potentially lose their data and have to set everything up all over again.

Any help would be greatly appreciated!

20

Update: In the end only one config file had a conffile prompt: /etc/janus/janus.jcfg.

After searching the internet I decided to chose Y to install the package maintainer’s version. However, I first made a backup by logging in to a new ssh session and copying the old config file.

I also had the issue that my /boot partition didn’t have enough free space to update initramfs. However, I solved this by running sudo apt autoremove, which seemed to free up enough space to proceed. I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again. Everything went well.

I then continued with steps (3)-(4) from Update - when to panic? - #17 by nbenedek.

I did NOT need to run step (5).

After rebooting I can log in via SSH. I still do not have access to the Plinth interface, but maybe it will need a few minutes to boot everything.

1 Like
21

Hi @dgj ,

I did all things above AND changed the firewalld-settings mentioned in https://discuss.freedombox.org/t/debian-12-bookworm-release-and-upgrading/2591/7.

It worked on two boxes.

Cheers Uli

1 Like
22

Yes, changing to “DefaultZone=external” in /etc/firewalld solved it.

I can access Plinth and ran Diagnostics. Everything has passed!

Thanks, everyone :slightly_smiling_face:

P.s. Nice to see some familiar names popping up in the forum again.

2 Likes
23

Yes, changing to “DefaultZone=external” in /etc/firewalld solved it.

I can access Plinth and ran Diagnostics. Everything has passed!

The change in /etc/firewalld/firewalld.conf of DefaultZone=external to DefaultZone=public seemed suspicious since, according to my notes, I had never touched that file. Choosing to accept the packager maintainer’s version when I attempted the manual upgrade (selecting Y at the conffile prompt) seems to have been the wrong choice for me. That said, I noticed that one of my FreedomBoxes that did successfully automatically upgrade without any issue (besides a minor Mediawiki hiccup) ended up with DefaultZone=external in its firewalld.conf file.

The box that had a successful automatic upgrade from Bullseye to Bookworm (originally spun up 2022-09-01) now has 3 files in /etc/firewalld/:

  • firewalld.conf with DefaultZone=external. Modified 2023-06-11.
  • firewalld.conf.old with DefaultZone=public. Modified 2023-01-06.
  • firewalld.conf.dpkg-old with DefaultZone=external. Modified 2022-08-17.

The FreedomBox that had the failed upgrade to from Bullseye to Bookworm (originally spun up on 2022-05-09) and which I reverted to an evening backup had in its backup image:

  • firewalld.conf with DefaultZone=external. Modified 2022-05-09.
  • firewalld.conf.old with DefaultZone=public. Modified 2021-02-01.

When I next attempt an upgrade, I’ll try making sure DefaultZone=external remains in /etc/firewalld/firewalld.conf.

24

donotd

2 Likes
25

I have now been without my Freedombox for four days. I feel I have tried most of the above (I am confused about this Firewall stuff - is that the original problem).

I just want it to work like it did on Saturday… been following the step by steps (I appreciate the suggestions) but I am getting this:

**matthew@freedombox**:**~**$ sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

Calculating upgrade... Done

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

1 not fully installed or removed.

After this operation, 0 B of additional disk space will be used.

Do you want to continue? [Y/n] y

Setting up initramfs-tools (0.142) ...

update-initramfs: deferring update (trigger activated)

Processing triggers for initramfs-tools (0.142) ...

update-initramfs: Generating /boot/initrd.img-6.1.0-9-armmp-lpae

W: Possible missing firmware /lib/firmware/imx/sdma/sdma-imx7d.bin for built-in driver imx_sdma

W: Possible missing firmware /lib/firmware/imx/sdma/sdma-imx6q.bin for built-in driver imx_sdma

zstd: error 70 : Write error : cannot write block : No space left on device

E: mkinitramfs failure zstd -q -9 -T0 70

update-initramfs: failed for /boot/initrd.img-6.1.0-9-armmp-lpae with 1.

dpkg: error processing package initramfs-tools (--configure):

installed initramfs-tools package post-installation script subprocess returned error exit status 1

Errors were encountered while processing:

initramfs-tools

E: Sub-process /usr/bin/dpkg returned an error code (1)

Any idea what I should do next?

My current thought is get rid of the Freedombox and go back to a hosted Matrix server or use Telegram and stop trying to be all ethical and open source - but hopefully that passes.