Since yesterday I cannot reach plinth, instead I get “The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.”
I can ping my Pioneer Freedombox, which is from 2019 an always updated.
Unfortunately I cannot reach it via ssh, because I have lost the password. All Password saved via KeePassXC do not work. Any hint for what I can do?
I have just seen, that the URL of my freedombox is “https://freedombox/plinth/sys/upgrades”. Does that mean, that I can not reach it, because it is upgrading? I learned from the forum, that upgrading takes a long time.
I hope that in the meantime you were able to fix things. I had the same and I after some days, I managed to resolve it. This assumes you have the cockpit app installed.
sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade. This started the installation. It took about 2 hours to upgrade. After that, I had to restart the RaspPi the hard wayI am seeing the same issue. I can’t even connect to the static default page and I’ve verified Apache is running and there is no firewall issue. I can connect via SSH so the system is up and running. Even trying to connect to the local IPv4 address fails.
Here is a clue:
netstat -tunlp | grep apache
tcp6 0 0 :::80 :::* LISTEN 1249/apache2
tcp6 0 0 :::443 :::* LISTEN 1249/apache2
I do not have IPv6 enabled on my LAN so the fact that it is not listening on IPv4 presents a problem.
My /etc/apache2/ports.conf looks reasonable:
# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf
Listen 80
<IfModule ssl_module>
Listen 443
</IfModule>
<IfModule mod_gnutls.c>
Listen 443
</IfModule>
Grepping for “ip” doesn’t reveal much:
grep -Rni "\bip" *
apache2.conf:119:# HostnameLookups: Log the names of clients or just their IP addresses
conf-available/email-server-freedombox.conf:16: RequestHeader unset X-Real-IP
mods-available/info.conf:8: #Require ip 192.0.2.0/24
mods-available/status.conf:8: #Require ip 192.0.2.0/24
At the moment I am stumped so I guess I do some Web searching to see just what might be going on.
As IPv6 was still enabled by the kernel, I edited /etc/default/flash-kernel as follows:
LINUX_KERNEL_CMDLINE="quiet ipv6.disable=1"
LINUX_KERNEL_CMDLINE_DEFAULTS=""
And I then ran dpkg-reconfigure linux-image-6.1.0-9-armmp-lpae and once that was complete, verified that the ipv6.disable=1 option had been added to /boot/boot.scr.
I then rebooted and verified that Apache is now listening on IPv4:
netstat -tunlp | grep apache
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1158/apache2
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1158/apache2
But I still have no access to the server.
I had used Aptitude via its UI earlier and it showed the apaches2 installation as being incomplete so it was completed when purging unneeded packages as there was a debconf file that needed my intervention for the installation to complete. Even so, things are still not working.
Still stumped.
Thank you all, but I do not see a solution.
Everything I’ve checked points to Apache simply not responding. I can monitor port 443 with tcpdump and I see the incoming traffic from my LAN and from public IPs. For whatever reason my Apache2 logs stopped back on 11 Nov 2022.
Compared to my other Linux systems, this one is the most vexing and problematic and has stopped being fun a long time ago. Even Quassel is failing since the Bookworm update. Other than SSH, the box is pretty much a paper weight right now, sad to say.
If not already done, you may want to check if the firewall is not dropping the traffic before it reaches apache. One way to do that is to reconfigure it to log all rejected packets (in /etc/firewalld/firewalld.conf LogDenied=all, then restart firewalld), try to access the web interface and check logs.
Turns out it was a firewall issue. Somewhere along the line the default zone was changed to “public” which only enabled SSH, fortunately. A custom zone file is in place named “external” that had been specified in an older version of firewalld.conf. Modifying firewalld.conf to DefaultZone=external and restarting showed that HTTPS was no longer being blocked.
Apache is accessible again. Thanks, Avron, for the tip. Now I can set about looking at other issues. Hopefully this helps @wromey.
With the firewall now working, Apache is dying about 30 seconds after being started. Perhaps someone quite more knowledgeable about Apache can help. I will note that things were working just fine prior to the upgrade to Bookworm.
The error that is causing Apache to die appears to be related to the ServerName:
Jun 20 07:17:11 freedombox apachectl[3250]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
I’ve tried various things to resolve this with no success. I have several names in /etc/hosts with the line for 127.0.1.1 and modified /etc/apache2/conf-enabled/freedombox-tls-site-macro.conf to set the ServerName directive to match one of those names, always without success as Apache finally leaves this message:
Jun 20 07:18:07 freedombox apache-error[3255]: [core:alert] [pid 3254:tid 3254] AH00050: Child 3260 returned a Fatal error... Apache is exiting!
Of course, the PID changes after each run.
Again, up until the system updated itself to Bookworm automatically this past Friday, the system was working just fine. Most annoying.
According to this link, you need to edit the /etc/apache2/apache2.conf file.
My guess, from many of the comments about the update, the dist-upgrade process whacks a lot of config files unexpectedly.
I suggest:
I have a similar problem.
Setting DefaultTimeoutStartSec=300s in mysettings.conf just keeps plinth running for longer before it disconnects. I tried setting DefaultTimeoutStartSec=600s in case some process needed more time, but the connection is lost after the time has elapsed. My Pioneer box is running bookworm with all packages updated, and ssh access works fine.
That was a useful tip as it cleared that particular error and for a short bit I can access the services such as plinth and Git Web and then Apache terminates again. On an off-chance I rebooted the box to see if something else is going on and apparently not as Apache is still terminating.
Here is the full log output:
Jun 21 06:11:41 freedombox systemd[1]: Started apache2.service - The Apache HTTP Server.
Jun 21 06:11:42 freedombox apache-error[1279]: [-:info] [pid 1248:tid 1248] mod_auth_pubtkt: version 0.13
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:info] [pid 1248:tid 1248] AH01883: Init: Initialized OpenSSL library
Jun 21 06:11:42 freedombox apache-error[1279]: [socache_shmcb:info] [pid 1248:tid 1248] AH00830: Shared memory socache initialised
Jun 21 06:11:42 freedombox apache-error[1279]: [socache_shmcb:info] [pid 1248:tid 1248] AH00830: Shared memory socache initialised
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:info] [pid 1248:tid 1248] AH01887: Init: Initializing (virtual) servers for SSL
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:info] [pid 1248:tid 1248] AH01914: Configuring server n0nb.freedombox.rocks:443 for SSL protocol
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:info] [pid 1248:tid 1248] AH02568: Certificate and private key n0nb.freedombox.rocks:443:0 configured from /etc/letsencrypt/live/n0nb.freedombox.rocks/fullchain.pem and /etc/letsencrypt/live/n0nb.freedombox.rocks/privkey.pem
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:info] [pid 1248:tid 1248] AH01914: Configuring server freedombox.lan:443 for SSL protocol
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:warn] [pid 1248:tid 1248] AH01909: freedombox.lan:443:0 server certificate does NOT include an ID which matches the server name
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:info] [pid 1248:tid 1248] AH02568: Certificate and private key freedombox.lan:443:0 configured from /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:error] [pid 1248:tid 1248] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=freedombox / issuer: CN=freedombox / serial: 65827D6267E6ADC2F0A399B6BC72287C8E5D2F25 / notbefore: May 4 02:05:10 2019 GMT / notafter: May 1 02:05:10 2029 GMT]
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:error] [pid 1248:tid 1248] AH02604: Unable to configure certificate freedombox.lan:443:0 for stapling
Jun 21 06:11:42 freedombox apache-error[1279]: [ssl:info] [pid 1248:tid 1248] AH01876: mod_ssl/2.4.57 compiled against Server: Apache/2.4.57, Library: OpenSSL/3.0.8
Jun 21 06:11:42 freedombox apache-error[1279]: [http2:info] [pid 1248:tid 1248] AH03090: mod_http2 (v2.0.11, feats=CHPRIO+SHA256+INVHD+DWINS, nghttp2 1.52.0), initializing...
Jun 21 06:11:42 freedombox apache-error[1279]: [http2:warn] [pid 1248:tid 1248] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
Jun 21 06:11:42 freedombox apache-error[1279]: [ldap:info] [pid 1248:tid 1248] AH01318: APR LDAP: Built with OpenLDAP LDAP SDK
Jun 21 06:11:42 freedombox apache-error[1279]: [ldap:info] [pid 1248:tid 1248] AH01319: LDAP: SSL support available
Jun 21 06:11:45 freedombox apache-error[1279]: [http2:info] [pid 1494:tid 1494] h2_workers: created with min=1 max=4 idle_ms=600000
Jun 21 06:11:45 freedombox apache-error[1279]: [ssl:info] [pid 1494:tid 1494] [client 185.191.225.130:59225] AH01964: Connection to child 0 established (server freedombox.lan:443)
Jun 21 06:11:45 freedombox apache-error[1279]: [http2:info] [pid 1506:tid 1506] h2_workers: created with min=1 max=4 idle_ms=600000
Jun 21 06:11:45 freedombox apache-error[1279]: [http2:info] [pid 1504:tid 1504] h2_workers: created with min=1 max=4 idle_ms=600000
Jun 21 06:11:45 freedombox apache-error[1279]: [core:info] [pid 1506:tid 1506] [client 185.191.225.130:48981] AH00128: File does not exist: /var/www/html/libera-proxy-checker.txt
Jun 21 06:11:45 freedombox apache-error[1279]: [http2:info] [pid 1508:tid 1508] h2_workers: created with min=1 max=4 idle_ms=600000
Jun 21 06:11:45 freedombox apache-access[1280]: freedombox.lan:80 185.191.225.130 - - [21/Jun/2023:06:11:45 -0500] "GET http://85.94.204.153/libera-proxy-checker.txt HTTP/1.0" 404 455 "-" "-"
Jun 21 06:11:45 freedombox apache-error[1279]: [http2:info] [pid 1514:tid 1514] h2_workers: created with min=1 max=4 idle_ms=600000
Jun 21 06:11:45 freedombox apache-access[1280]: freedombox.lan:80 185.191.225.130 - - [21/Jun/2023:06:11:45 -0500] "POST http://85.94.204.153/libera-proxy-checker.txt HTTP/1.0" 404 455 "-" "-"
Jun 21 06:11:45 freedombox apache-error[1279]: [core:info] [pid 1504:tid 1504] [client 185.191.225.130:53937] AH00128: File does not exist: /var/www/html/libera-proxy-checker.txt
Jun 21 06:11:45 freedombox apache-access[1280]: freedombox.lan:80 185.191.225.130 - - [21/Jun/2023:06:11:45 -0500] "CONNECT 85.94.204.153:31204 HTTP/1.0" 302 522 "-" "-"
Jun 21 06:11:45 freedombox apache-error[1279]: [mpm_prefork:notice] [pid 1248:tid 1248] AH00163: Apache/2.4.57 (Debian) mod_auth_pubtkt/0.13 OpenSSL/3.0.9 configured -- resuming normal operations
Jun 21 06:11:45 freedombox apache-error[1279]: [mpm_prefork:info] [pid 1248:tid 1248] AH00164: Server built: 2023-04-13T03:26:51
Jun 21 06:11:45 freedombox apache-error[1279]: [core:notice] [pid 1248:tid 1248] AH00094: Command line: '/usr/sbin/apache2'
Jun 21 06:11:45 freedombox apache-access[1280]: freedombox.lan:80 185.191.225.130 - - [21/Jun/2023:06:11:45 -0500] "GET / HTTP/1.1" 302 452 "-" "-"
Jun 21 06:13:35 freedombox apache-error[1279]: [mpm_prefork:emerg] [pid 1514:tid 1514] (43)Identifier removed: AH00144: couldn't grab the accept mutex
Jun 21 06:13:35 freedombox apache-error[1279]: [mpm_prefork:emerg] [pid 1504:tid 1504] (43)Identifier removed: AH00144: couldn't grab the accept mutex
Jun 21 06:13:35 freedombox apache-error[1279]: [mpm_prefork:emerg] [pid 1508:tid 1508] (43)Identifier removed: AH00144: couldn't grab the accept mutex
Jun 21 06:13:35 freedombox apache-error[1279]: [mpm_prefork:emerg] [pid 1494:tid 1494] (43)Identifier removed: AH00144: couldn't grab the accept mutex
Jun 21 06:13:36 freedombox apache-error[1279]: [core:alert] [pid 1248:tid 1248] AH00050: Child 1494 returned a Fatal error... Apache is exiting!
Jun 21 06:13:36 freedombox apache-error[1279]: [Wed Jun 21 06:13:36.559080 2023] [:emerg] [pid 1248] AH02818: MPM run failed, exiting
I may be mistaken but it appears that SSL certificates, even though n0nb.freedombox.rocks has a Let’s Encrypt certificate, are the cause of this error. In my particular setup with my Freedombox set up behinds a router/firewall and with no desire to expose it to the broader Internet, SSL gets in the way as Let’s Encrypt demands that both ports 80 and 443 be open for certificate renewal whenever that happens. I tried to do without SSL earlier this year but ran into problems with one or more of the modules.
Sigh…
The same message appeared here and there and in both cases went away following actions that did not touch Apache or certificate configuration.
That makes me think that it might not be a problem of Apache configuration, rather that something else is interfering with Apache (and the “couln’t grab mutex” may be a symptom of that?). I did not understand the exact role of the timer, you could try to put it to a longer value. Or you could try to remove a package (rather a “big” one).
I agree with your reading of the logs that it looks like the SSL certificate is to blame.
I’ll be honest and say I’m not the greatest with SSL. Its not quite voodoo magic for me but I’ve had general success for the most part with getting issued certs and so I do not have much experience troubleshooting and definitely not with Apache2.
freedombox.lan:443:0 server certificate does NOT include an ID which matches the server name
Sounds like the stated server name doesn’t match what the cert is for. I’m thinking you had a borked fetch at some point early on and you are dealing with the fall out.
For this error, I found this guide - at the least, check spots indicated and see what the file says but don’t change anything yet as the next error might be the better path to tackle.
ssl_stapling_init_cert: can’t retrieve issuer certificate!
Sounds like it can’t retrieve LC’s cert.
This seems like a way to kill SSL temporarily and maybe get things going to the point you can trigger a new LC cert pull.
My problems were caused by a faulty network setup - I had an ethernet connection, but no internal WAN. Tried to edit the connection manually, but wasn’t able to make it work properly. Reinstalled using the nightly image, everything seems to be working now.
Avron is on the right track. I searched for the mutex exception message and was led to:
The very last comment that suggested adding Mutex posixsem to the apache2.conf which I did and it now has been running for nearly a half hour.
Now I need to get tt-rss going. it seems as though the upgrade to postgresql to version 15 didn’t migrate the database. Sigh.
My database was even more worse off than simply not being migrated. It was still in Postgreql 11 from Buster! To make matters worse I had managed to delete the old database. Fortunately, I had a copy that was about three months old and copied it over and then installed PG 11 from the Buster archive. The tt-rss application is working as is Quassel. I will need to do the DB migration, but aleast the pieces are working again.
Thanks for all the help! I stayed away from this for several days and that helped me a lot.