Update - when to panic?

Realized I should have posted this in Pioneerbox support - apologies.

Yesterday I noticed that Fredombox was updating. Today, I could not login to plinth, element not connecting to Matrix and none of the web interface is working.

I have not been able to find anything through a forum search that I should try.

  1. I can login through SSH.
  2. I cannot connect through the dynamic DNS address on a browser.
  3. I cannot connect through the IP address of the box, or freedombox.local

Should I panic?

Anything I should do to check or will it just sort itself out?

I did a hard reboot this morning (may have been a mistake) and it did not resolve anything.

Although I can SSH and follow basic instructions I am not a linux expert so I love step by step instructions :slight_smile:

I am in almost the exact same situation, ssh access, no access otherwise. I also rebooted today when I logged in via ssh.

I am comfortable with the command line and see that there are manual update instructions at the FreedomBox manual:

https://wiki.debian.org/FreedomBox/Manual#FreedomBox.2FManual.2FUpgrades.Software_Updates

Any suggestions? Would a manual update be recommended at this point?

Don’t panic - release upgrades are huge and may take a day or more to complete. Try to give your Freedombox time to grind through it. At least give it a day if you can, and then try to look for problems once the upgrade finishes.

Update - plinth is down for me on the 11th at 08:00. I’m going to wait it out. I’ll check in when all of the plinth dependencies are upgraded in hours.

Is there a way to confirm from the terminal if an upgrade is currently running?

1 Like

On my side:

  • plinth gives error 503
  • trying to ssh gets “connection refused”

The “connection refused” gets me worried as I basically have no way to see what is happening.

EDIT: I managed to connect. I rebooted, ejabberd, matrix and plinth fail to start. I am looking into /var/log/apt/history.log, it ends up with:

Start-Date: 2023-06-11 08:26:23
Commandline: /usr/bin/unattended-upgrade
Upgrade: btrfs-progs:armhf (5.10.1-2, 6.2-1~bpo11+1)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2023-06-11 08:30:15

Start-Date: 2023-06-11 08:33:01
Commandline: /usr/bin/unattended-upgrade
Remove: python3.9:armhf (3.9.2-1), python3.9-minimal:armhf (3.9.2-1)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2023-06-11 08:37:57

Looks like some update failed and maybe it remained stuck? Postfix and dovecot work.

EDIT 2: “apt -s upgrade” says there are 706 packages to update, 109 new and 8 not updated. I guess I’ll try doing the upgrade manually.

If you have SSH access, you can issue the following commands to get some info about the update process: sudo journalctl --follow --unit=plinth.service and sudo tail -f /var/log/*

On my side, this command gives:

juin 11 18:24:37 fbox systemd[1]: plinth.service: Consumed 11.456s CPU time.
juin 11 18:24:43 fbox systemd[1]: plinth.service: Scheduled restart job, restart counter is at 48.
juin 11 18:24:43 fbox systemd[1]: Stopped plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:43 fbox systemd[1]: plinth.service: Consumed 11.456s CPU time.
juin 11 18:24:43 fbox systemd[1]: Started plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:54 fbox systemd[1]: plinth.service: Main process exited, code=exited, status=1/FAILURE
juin 11 18:24:54 fbox systemd[1]: plinth.service: Failed with result ‘exit-code’.
juin 11 18:24:54 fbox systemd[1]: plinth.service: Consumed 11.373s CPU time.
juin 11 18:24:55 fbox systemd[1]: Stopped plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:55 fbox systemd[1]: plinth.service: Consumed 11.373s CPU time.

I ran “systemctl stop plinth” (same for ejabberd) since that thing is repeating itself and brings nothing.

I have the same 503 error, just posting to confirm there is an issue.

I used a spare Micro Sd Card and a fresh installation, started the update via terminal, now going to wait a bit and see what happens.

When I run “apt update”, I have:

Hit:1 Index of /debian stable InRelease
Get:2 Index of /debian stable-updates InRelease [52.1 kB]
Get:3 Index of /debian bullseye-backports InRelease [49.0 kB]
Get:4 Index of /debian-security stable-security InRelease [48.0 kB]
Get:5 Index of /debian-security stable-security/main Sources [9184 B]
Get:6 Index of /debian-security stable-security/main armhf Packages [17.2 kB]
Get:7 Index of /debian-security stable-security/main Translation-en [7460 B]

Is it ok to have bullseye-backports when stable is now bookworm? (that comes from /etc/apt/sources.list.d/freedombox2.list). Should I remove that file before upgrading?

I found out that /boot is full and this is why update-initramfs fails.

/boot seems to include 4 initrd.img and 4 vmlinuz. (5.10.0-10/22/23 and 6.1.0.0-0). “uname -a” says that 6.1.0-0 is used so perhaps I can just remove 5.10.0-10/22 to make space?

I thought I’d do the same, but ended up with a server failing to serve any web pages (including the Plinth webUI) despite the apache2 service running. I had to restore from an evening snapshot of the server (an AWS Lightsail snapshot, not the FreedomBox backup).

During the manual upgrade ($ sudo apt update && sudo apt upgrade) I had to answer several prompts involving LDAP-related prompts (nslcd, libnss-ldapd) for which I just hit ‘Enter’, accepting the defaults, and hoped for the best. Some of these changes involved:

  • Configuring nslcd
    • LDAP sever URI : I accepted the default ldapi:/// .
  • Configuring nslcd
    • LDAP server search base: I accepted the default dc=thisbox .
  • Configuring popularity-contest
    • Participate in the package usage survey?: I accepted the default <No>.
  • Configuring libnss-ldapd
    • Name services to configure:: I kept the default choices selected:
      • [*] passwd
      • [*] group
      • [*] shadow
      • Note: Various error messages following the failed upgrade appear in journalctl logs mention nslcd and passwd which weren’t present in such logs prior to the upgrade attempt. I don’t know what to make of them.

I did get prompts for accepting or rejecting changes to configuration files I’d modified:

  • /etc/ssh/sshd_config. I accepted the package maintainer version. Some changes I noticed were:
    • PasswordAuthentication no: to disable password authentication, a change I remember making.
    • StreamLocalBindUnlink yes: to enable GnuPG Agent Forwarding; not essential to login via SSH; the change was one I made for the convenience of signing git commits with my local OpenPGP smartcard.
    • TrustedUserCAKeys /etc/ssh/lightsail_instance_ca.pub: Not my modification; a change I believe was already present in the Debian image I booted the AWS Lightsail instance from before installing the freedombox package back in 2022-05.
  • /etc/firewalld/firewalld.conf: I don’t believe I had ever touched this, so I just accepted the package maintainer version which contained, among other changes, a removal of AllowZoneDrifting=no and change of DefaultZone=external to DefaultZone=public.

At some point, failing to see the package freedombox get upgraded due to a hold, I decided to risk running $ sudo apt-mark unhold freedombox && sudo apt upgrade -y; I only did this because I had a recent backup of the entire server; the result was the Plinth webUI failing to start, and no apache2 pages being served (despite the # systemctl status apache2.service showing the service was running) despite being able to login via SSH.

After about half an hour of not seeing any significant CPU work, I decided to restore from an evening snapshot taken by Lightsail prior to the update; after starting the snapshot in a new machine instance, I unchecked the “Enable auto-update to next stable release” option in the System > Software Update section; now I’m waiting to see dust settle here on this forum before I make another upgrade attempt.

1 Like

I tried running sudo journalctl --follow --unit=plinth.service and I am getting the same error message.

Unfortunately I do not have a recent backup and thus will wait if there is any help for upgrading safely and without needing to answer any prompts.

Help much appreciated. My family is now regularly using the FreedomBox and they are feeling the downtime.

1 Like

I’ll try to reproduce the issue, I’ll get back with my findings.

I ran

$ sudo su -
Password: <enter user password here>
# dpkg --configure -a
# apt update
# apt -f install
# unattended-upgrade --debug
# apt install freedombox
# apt update
# apt full-upgrade

After reboot, trying to reach the web interface gives an error that the server is not secure.

I have the following problem with apache2:

root@fbox:/etc/apache2# journalctl --follow --unit=apache2.service
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:warn] [pid 1113:tid 1113] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox systemd[1]: Started apache2.service - The Apache HTTP Server.
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:warn] [pid 1167:tid 1167] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox apache-error[1168]: [http2:warn] [pid 1167:tid 1167] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
Jun 11 21:58:17 fbox apache-error[1168]: [mpm_prefork:notice] [pid 1167:tid 1167] AH00163: Apache/2.4.57 (Debian) mod_auth_pubtkt/0.13 OpenSSL/3.0.9 configured -- resuming normal operations
Jun 11 21:58:17 fbox apache-error[1168]: [core:notice] [pid 1167:tid 1167] AH00094: Command line: '/usr/sbin/apache2'

I replaced my domain name with “mydomain.tld”. To reach my freedombox, I only type “https://mydomain.eu” not “https://fbox.mydomain.eu”. I don’t know whether this has anything to do with the issue (it used to work before upgrade).

EDIT: I did the following:

Apparently, ejabberd and dovecot are running but they are unreachable :frowning:

Broke my Pioneer box.

My AMD btrfs server is up for previous services, but
WireGuard is broken.

SearX was removed as an installed app.

  • S

I am able to access tt-rss via my phone app but the VPN is like 80% down. Plinth is also down with the following error for like 1 day right now. A restart didn’t resolve it. Should I just wait? I don’t have ssh access.

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.


Apache/2.4.56 (Debian) Server at freedombox.local Port 443

I was able to reproduce a failed upgrade on a Raspberry Pi 4. The FreedomBox web interface became unavailable, being stuck in the same error loop described above by @Avron. Here’s what you can do if you have SSH or console access:
0. (if you use SSH) install screen, to stay connected even if the SSH session disconnects:
sudo apt install -y screen

  1. screen [enter, then press space]
  2. sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  3. after the upgrade finished, I ran apt update again, and saw that the freedombox package was held, so ran: sudo apt-mark unhold freedombox
  4. Then I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again.
  5. Now my /boot partition didn’t have enough free space to update initramfs. If you are facing the same issue, you can free up some space and then update initramfs. Please only do this if update-initramfs is failing for you:
    5.1. sudo mkdir /root/kernel-backup
    5.2. sudo mv /boot/firmware/initrd.img-5* /root/kernel-backup/
    5.3. sudo mv /boot/firmware/vmlinuz-5* /root/kernel-backup/
    5.4 DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
  6. sudo reboot
1 Like

I am at step (2) but now it is asking me about config files, and I do not want to make the wrong choice and break my setup.

Any suggestions for what to do when it asks:

Configuration file X
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.

Note that I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade and did not expect to be asked about config files.

Edited to add: I see I already asked about the “noninteractive” option here: Unattended-upgrades vs. DEBIAN_FRONTEND=noninteractive apt-get upgrade vs. apt-get upgrade -y

From the above link re the “noninteractive” prompt, I see this answer from @jvalleroy:

unattended-upgrades will not upgrade packages that have a conffile prompt. This is a good thing, because we often release new versions of FreedomBox to properly handle conffile prompts. FreedomBox will check for packages to be upgraded on a regular basis.

Does this mean I am already in a bad situation because I am upgrading a package with a conffile prompt? Several people in my family are now using the box and I really do not want to have to do a fresh install and potentially lose their data and have to set everything up all over again.

Any help would be greatly appreciated!

Update: In the end only one config file had a conffile prompt: /etc/janus/janus.jcfg.

After searching the internet I decided to chose Y to install the package maintainer’s version. However, I first made a backup by logging in to a new ssh session and copying the old config file.

I also had the issue that my /boot partition didn’t have enough free space to update initramfs. However, I solved this by running sudo apt autoremove, which seemed to free up enough space to proceed. I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again. Everything went well.

I then continued with steps (3)-(4) from Update - when to panic? - #17 by nbenedek.

I did NOT need to run step (5).

After rebooting I can log in via SSH. I still do not have access to the Plinth interface, but maybe it will need a few minutes to boot everything.

1 Like