Hi folks,
I haven’t been keeping a close eye on my FreedomBox for quite some time, I’ve just been letting it do its thing. I don’t have too much running right now–just Syncthing, miniDLNA, bepasty and Searx–and everything has been working fine. I have noticed the web portal has been awfully slow but haven’t thought much of it.
I was tidying up the area over by the FreedomBox today and discovered the unit is blazing hot. I opened up the Cockpit app to check if there was some kind of temperature gauge I could take a look at, to see if perhaps I was overreacting. I didn’t see a temperature gauge, but I did see the CPU is maxed out at 100%:
A service called uwsgi is holding down some 95% CPU usage. I looked it up quickly, the software has something to do with building hosting services so I guess that seems important–but also the CPU usage seems like maybe it is not working properly.
I restarted the service from Cockpit, but it immediately shot right back to 95+% CPU usage. I shut the box down completely and left it off for the better part of an hour, but as soon as it was booted back up CPU usage shot back up to 100% and stayed there.
Does anyone have any insight into what I can do to troubleshoot this uwsgi service?
Information
Debian GNU/Linux 11 (bullseye) and FreedomBox version 22.10
Hardware
Olimex A20-OLinuXino-LIME2 board
How did you install FreedomBox?
Purchased pre-installed version.
I decided to stop the uwsgi service from Cockpit and see what happens. The CPU usage dropped right off of course, and all of a sudden the web FreedomBox web interface was very snappy and responsive. I checked on the apps and discovered Syncthing and miniDLNA were both up and seemed fine but bepasty and Searx were both broken.
I toggled bepasty off and on again in the app menu and it came right back up, but this restarted the uwsgi service and CPU usage immediately shot up to 100% and stuck there.
For now I’m just going to leave Searx and bepasty off so I can keep the uwsgi service stopped until I figure out how to get it back under control.
You can check the log files under /var/log/uwsgi/app. I would guess one of the apps is either stuck, or continuously restarting.
Aha–thanks for the tip James, looks like you are absolutely right. I have a very large searx.log in here, with this recurring error message:
GNU nano 5.4 searx.log
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:16 2022 - DAMN ! worker 3 (pid: 10933) died :( trying respawn ...
Sun May 15 00:09:16 2022 - Respawned uWSGI worker 3 (new pid: 10993)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:17 2022 - DAMN ! worker 4 (pid: 10932) died :( trying respawn ...
Sun May 15 00:09:17 2022 - Respawned uWSGI worker 4 (new pid: 10996)
Sun May 15 00:09:17 2022 - DAMN ! worker 2 (pid: 10934) died :( trying respawn ...
Sun May 15 00:09:17 2022 - Respawned uWSGI worker 2 (new pid: 10997)
Sun May 15 00:09:17 2022 - DAMN ! worker 1 (pid: 10935) died :( trying respawn ...
Sun May 15 00:09:17 2022 - Respawned uWSGI worker 1 (new pid: 10998)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:26 2022 - DAMN ! worker 3 (pid: 10993) died :( trying respawn ...
Sun May 15 00:09:26 2022 - Respawned uWSGI worker 3 (new pid: 11004)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:29 2022 - DAMN ! worker 4 (pid: 10996) died :( trying respawn ...
Sun May 15 00:09:29 2022 - Respawned uWSGI worker 4 (new pid: 11005)
Sun May 15 00:09:29 2022 - DAMN ! worker 2 (pid: 10997) died :( trying respawn ...
Sun May 15 00:09:29 2022 - Respawned uWSGI worker 2 (new pid: 11006)
Sun May 15 00:09:29 2022 - DAMN ! worker 1 (pid: 10998) died :( trying respawn ...
Sun May 15 00:09:29 2022 - Respawned uWSGI worker 1 (new pid: 11007)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:36 2022 - DAMN ! worker 3 (pid: 11004) died :( trying respawn ...
Sun May 15 00:09:36 2022 - Respawned uWSGI worker 3 (new pid: 11010)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:39 2022 - DAMN ! worker 4 (pid: 11005) died :( trying respawn ...
Sun May 15 00:09:39 2022 - Respawned uWSGI worker 4 (new pid: 11012)
Sun May 15 00:09:39 2022 - DAMN ! worker 2 (pid: 11006) died :( trying respawn ...
Sun May 15 00:09:39 2022 - Respawned uWSGI worker 2 (new pid: 11013)
Sun May 15 00:09:39 2022 - DAMN ! worker 1 (pid: 11007) died :( trying respawn ...
Sun May 15 00:09:39 2022 - Respawned uWSGI worker 1 (new pid: 11014)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:45 2022 - DAMN ! worker 3 (pid: 11010) died :( trying respawn ...
Sun May 15 00:09:45 2022 - Respawned uWSGI worker 3 (new pid: 11017)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:48 2022 - DAMN ! worker 4 (pid: 11012) died :( trying respawn ...
Sun May 15 00:09:48 2022 - Respawned uWSGI worker 4 (new pid: 11018)
Sun May 15 00:09:48 2022 - DAMN ! worker 2 (pid: 11013) died :( trying respawn ...
Sun May 15 00:09:48 2022 - Respawned uWSGI worker 2 (new pid: 11019)
Sun May 15 00:09:48 2022 - DAMN ! worker 1 (pid: 11014) died :( trying respawn ...
Sun May 15 00:09:48 2022 - Respawned uWSGI worker 1 (new pid: 11020)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:53 2022 - DAMN ! worker 3 (pid: 11017) died :( trying respawn ...
Sun May 15 00:09:53 2022 - Respawned uWSGI worker 3 (new pid: 11022)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:09:57 2022 - DAMN ! worker 4 (pid: 11018) died :( trying respawn ...
Sun May 15 00:09:57 2022 - Respawned uWSGI worker 4 (new pid: 11028)
Sun May 15 00:09:57 2022 - DAMN ! worker 2 (pid: 11019) died :( trying respawn ...
Sun May 15 00:09:57 2022 - Respawned uWSGI worker 2 (new pid: 11029)
Sun May 15 00:09:57 2022 - DAMN ! worker 1 (pid: 11020) died :( trying respawn ...
Sun May 15 00:09:57 2022 - Respawned uWSGI worker 1 (new pid: 11030)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:10:02 2022 - DAMN ! worker 3 (pid: 11022) died :( trying respawn ...
Sun May 15 00:10:02 2022 - Respawned uWSGI worker 3 (new pid: 11037)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:10:05 2022 - DAMN ! worker 2 (pid: 11029) died :( trying respawn ...
Sun May 15 00:10:05 2022 - Respawned uWSGI worker 2 (new pid: 11040)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:10:06 2022 - DAMN ! worker 4 (pid: 11028) died :( trying respawn ...
Sun May 15 00:10:06 2022 - Respawned uWSGI worker 4 (new pid: 11041)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:10:07 2022 - DAMN ! worker 1 (pid: 11030) died :( trying respawn ...
Sun May 15 00:10:07 2022 - Respawned uWSGI worker 1 (new pid: 11042)
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
Sun May 15 00:10:11 2022 - DAMN ! worker 3 (pid: 11037) died :( trying respawn ...
That’s just a snippet, it goes on like that for pages and pages.
Recently I made an effort to modify the Searx instance by copying the /etc/searx/settings.yml off of the Searx GitHub page. I handily broke Searx with this move, but at the time I just left it thinking I would double-back and try to fix it later. I never did; I wound up setting up Whoogle on another box and kind of forgot about Searx.
Case closed–thanks again for the tip James. * tips hat *