Realized I should have posted this in Pioneerbox support - apologies.
Yesterday I noticed that Fredombox was updating. Today, I could not login to plinth, element not connecting to Matrix and none of the web interface is working.
I have not been able to find anything through a forum search that I should try.
I can login through SSH.
I cannot connect through the dynamic DNS address on a browser.
I cannot connect through the IP address of the box, or freedombox.local
Should I panic?
Anything I should do to check or will it just sort itself out?
I did a hard reboot this morning (may have been a mistake) and it did not resolve anything.
Although I can SSH and follow basic instructions I am not a linux expert so I love step by step instructions
Donāt panic - release upgrades are huge and may take a day or more to complete. Try to give your Freedombox time to grind through it. At least give it a day if you can, and then try to look for problems once the upgrade finishes.
Update - plinth is down for me on the 11th at 08:00. Iām going to wait it out. Iāll check in when all of the plinth dependencies are upgraded in hours.
If you have SSH access, you can issue the following commands to get some info about the update process: sudo journalctl --follow --unit=plinth.service and sudo tail -f /var/log/*
juin 11 18:24:37 fbox systemd[1]: plinth.service: Consumed 11.456s CPU time.
juin 11 18:24:43 fbox systemd[1]: plinth.service: Scheduled restart job, restart counter is at 48.
juin 11 18:24:43 fbox systemd[1]: Stopped plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:43 fbox systemd[1]: plinth.service: Consumed 11.456s CPU time.
juin 11 18:24:43 fbox systemd[1]: Started plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:54 fbox systemd[1]: plinth.service: Main process exited, code=exited, status=1/FAILURE
juin 11 18:24:54 fbox systemd[1]: plinth.service: Failed with result āexit-codeā.
juin 11 18:24:54 fbox systemd[1]: plinth.service: Consumed 11.373s CPU time.
juin 11 18:24:55 fbox systemd[1]: Stopped plinth.service - FreedomBox Service (Plinth).
juin 11 18:24:55 fbox systemd[1]: plinth.service: Consumed 11.373s CPU time.
I ran āsystemctl stop plinthā (same for ejabberd) since that thing is repeating itself and brings nothing.
Is it ok to have bullseye-backports when stable is now bookworm? (that comes from /etc/apt/sources.list.d/freedombox2.list). Should I remove that file before upgrading?
I found out that /boot is full and this is why update-initramfs fails.
/boot seems to include 4 initrd.img and 4 vmlinuz. (5.10.0-10/22/23 and 6.1.0.0-0). āuname -aā says that 6.1.0-0 is used so perhaps I can just remove 5.10.0-10/22 to make space?
I thought Iād do the same, but ended up with a server failing to serve any web pages (including the Plinth webUI) despite the apache2 service running. I had to restore from an evening snapshot of the server (an AWS Lightsail snapshot, not the FreedomBox backup).
During the manual upgrade ($ sudo apt update && sudo apt upgrade) I had to answer several prompts involving LDAP-related prompts (nslcd, libnss-ldapd) for which I just hit āEnterā, accepting the defaults, and hoped for the best. Some of these changes involved:
Configuring nslcd
LDAP sever URI : I accepted the default ldapi:/// .
Configuring nslcd
LDAP server search base: I accepted the default dc=thisbox .
Configuring popularity-contest
Participate in the package usage survey?: I accepted the default <No>.
Configuring libnss-ldapd
Name services to configure:: I kept the default choices selected:
[*] passwd
[*] group
[*] shadow
Note: Various error messages following the failed upgrade appear in journalctl logs mention nslcd and passwd which werenāt present in such logs prior to the upgrade attempt. I donāt know what to make of them.
I did get prompts for accepting or rejecting changes to configuration files Iād modified:
/etc/ssh/sshd_config. I accepted the package maintainer version. Some changes I noticed were:
PasswordAuthentication no: to disable password authentication, a change I remember making.
StreamLocalBindUnlink yes: to enable GnuPG Agent Forwarding; not essential to login via SSH; the change was one I made for the convenience of signing git commits with my local OpenPGP smartcard.
TrustedUserCAKeys /etc/ssh/lightsail_instance_ca.pub: Not my modification; a change I believe was already present in the Debian image I booted the AWS Lightsail instance from before installing the freedombox package back in 2022-05.
/etc/firewalld/firewalld.conf: I donāt believe I had ever touched this, so I just accepted the package maintainer version which contained, among other changes, a removal of AllowZoneDrifting=no and change of DefaultZone=external to DefaultZone=public.
At some point, failing to see the package freedombox get upgraded due to a hold, I decided to risk running $ sudo apt-mark unhold freedombox && sudo apt upgrade -y; I only did this because I had a recent backup of the entire server; the result was the Plinth webUI failing to start, and no apache2 pages being served (despite the # systemctl status apache2.service showing the service was running) despite being able to login via SSH.
After about half an hour of not seeing any significant CPU work, I decided to restore from an evening snapshot taken by Lightsail prior to the update; after starting the snapshot in a new machine instance, I unchecked the āEnable auto-update to next stable releaseā option in the System > Software Update section; now Iām waiting to see dust settle here on this forum before I make another upgrade attempt.
After reboot, trying to reach the web interface gives an error that the server is not secure.
I have the following problem with apache2:
root@fbox:/etc/apache2# journalctl --follow --unit=apache2.service
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:warn] [pid 1113:tid 1113] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:11 fbox apache-error[1163]: [ssl:error] [pid 1113:tid 1113] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox systemd[1]: Started apache2.service - The Apache HTTP Server.
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:warn] [pid 1167:tid 1167] AH01909: fbox.mydomain.tld:443:0 server certificate does NOT include an ID which matches the server name
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 64EB79AF9D45607D587781C5FC38634604FB6FDA / notbefore: Jan 29 11:08:46 2023 GMT / notafter: Jan 26 11:08:46 2033 GMT]
Jun 11 21:58:13 fbox apache-error[1168]: [ssl:error] [pid 1167:tid 1167] AH02604: Unable to configure certificate fbox.mydomain.tld:443:0 for stapling
Jun 11 21:58:13 fbox apache-error[1168]: [http2:warn] [pid 1167:tid 1167] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
Jun 11 21:58:17 fbox apache-error[1168]: [mpm_prefork:notice] [pid 1167:tid 1167] AH00163: Apache/2.4.57 (Debian) mod_auth_pubtkt/0.13 OpenSSL/3.0.9 configured -- resuming normal operations
Jun 11 21:58:17 fbox apache-error[1168]: [core:notice] [pid 1167:tid 1167] AH00094: Command line: '/usr/sbin/apache2'
I replaced my domain name with āmydomain.tldā. To reach my freedombox, I only type āhttps://mydomain.euā not āhttps://fbox.mydomain.euā. I donāt know whether this has anything to do with the issue (it used to work before upgrade).
change owner of .pem files in /etc/ejabberd/ to ejabberd instead of root (because in syslog there was a failure to read it), now ejabberd says it starts
Apparently, ejabberd and dovecot are running but they are unreachable
I am able to access tt-rss via my phone app but the VPN is like 80% down. Plinth is also down with the following error for like 1 day right now. A restart didnāt resolve it. Should I just wait? I donāt have ssh access.
Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.4.56 (Debian) Server at freedombox.local Port 443
I was able to reproduce a failed upgrade on a Raspberry Pi 4. The FreedomBox web interface became unavailable, being stuck in the same error loop described above by @Avron. Hereās what you can do if you have SSH or console access:
0. (if you use SSH) install screen, to stay connected even if the SSH session disconnects: sudo apt install -y screen
after the upgrade finished, I ran apt update again, and saw that the freedombox package was held, so ran: sudo apt-mark unhold freedombox
Then I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again.
Now my /boot partition didnāt have enough free space to update initramfs. If you are facing the same issue, you can free up some space and then update initramfs. Please only do this if update-initramfs is failing for you:
5.1. sudo mkdir /root/kernel-backup
5.2. sudo mv /boot/firmware/initrd.img-5* /root/kernel-backup/
5.3. sudo mv /boot/firmware/vmlinuz-5* /root/kernel-backup/
5.4 DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
I am at step (2) but now it is asking me about config files, and I do not want to make the wrong choice and break my setup.
Any suggestions for what to do when it asks:
Configuration file X
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
Note that I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade and did not expect to be asked about config files.
From the above link re the ānoninteractiveā prompt, I see this answer from @jvalleroy:
unattended-upgrades will not upgrade packages that have a conffile prompt. This is a good thing, because we often release new versions of FreedomBox to properly handle conffile prompts. FreedomBox will check for packages to be upgraded on a regular basis.
Does this mean I am already in a bad situation because I am upgrading a package with a conffile prompt? Several people in my family are now using the box and I really do not want to have to do a fresh install and potentially lose their data and have to set everything up all over again.
Update: In the end only one config file had a conffile prompt: /etc/janus/janus.jcfg.
After searching the internet I decided to chose Y to install the package maintainerās version. However, I first made a backup by logging in to a new ssh session and copying the old config file.
I also had the issue that my /boot partition didnāt have enough free space to update initramfs. However, I solved this by running sudo apt autoremove, which seemed to free up enough space to proceed. I ran sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade again. Everything went well.
