System unstable after upgrade to bullseye

Boom, same procedure, now after 10 days, anyhow. But it IS frustrating, because it happens for no apparent reason, and I am getting aware of the system being down only after I can’t chat anymore or commit to my git service, sometimes when being offroad and NOT being able to just hard resetting the box.
I also wonder if I should change the OS.
But I’d prefer to have this issue fixed ASAP, because I’m quite happy with my setup were it not for the casual hangs. Everything else means a probably time consuming process of reinstalling and reconfiguring what I’ve already done.

Is it normal that, at boot, the logs don’t show the same long sequence of kernel messages?

In my own logs, except when I had serious problems, I usually see something starting with:

kernel: Booting Linux on physical CPU 0x0
kernel: Linux version 5.10.0-9-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)
kernel: CPU: ARMv7 Processor [410fc074] revision 4 (ARMv7), cr=30c5387d

At my system, this is the journal log when filtering for those kernel messages:

oliver@freedombox:~$ sudo journalctl -q -g "Linux version"
Failed to get MESSAGE field: Bad message
Jul 13 17:29:26 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Jul 13 17:29:26 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Jul 13 17:29:26 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Jul 13 17:29:26 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Jul 13 17:29:26 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Jul 13 17:29:26 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Jul 13 17:29:26 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Aug 18 02:00:52 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>
Aug 18 09:19:41 freedombox kernel: Linux version 5.10.0-8-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.>

As @sunil pointed out above, the identical time entries may stem from System clock time unset or jumped backwards, because there is no battery backed RTC clock.

There are no messages saying Linux version ... anymore after Dist-Upgrade to Bullseye, or so it seems. How is that?

We have to thank to Olimex team for updating the images of Bulseye properly. Thanks to them, the system now works stably.

Hello Johnny,
Is it possible to reinstall the new freedombox image with the fixed Bulseye over an existing installation? Or do we just reinstall the system from scratch?
Dave

Hi, Dave! The system was unstable after the upgrade with Bullseye, and this went on for a long time without being fixed. So I decided to contact Olimex about this. They replied that they were not aware of this problem and would check what they could do about it. Their reaction was lightning fast - on the same day they had uploaded the images with a properly updated bullseye and wrote to me that they would contact the Freedombox team. Without having to reinstall it, my pioneer started working stably after the last update, so I guess the problem is solved. Do you have any problems with yours?

1 Like

Hi Johnny, After upgrade to Bullseye, the system would crash and become unreachable from the outside world (web, ssh, etc.) every 1 to 3 days. Thinking that it might be hardware related, I purchased a new Olimex A20 Lime 2 board (Rev. L I believe) in December, installed the weekly FB stable image, and it too crashed after a few days. I can’t say if the latest update fixed the problem or not because I just wanted to get it running again after reading your post that Olimex issued a new Bullseys version. Only had Samba, OpenVPN, and Apache running, so it certainly was not overloaded, and the SD card was not filled with Snapshots. Just installed the latest version today and am waiting to see if it is now stable. Thanks for your fast reply.

Hello Oliver,
If you don’t already know, about three days ago, Olimex updated their Bullseye FreedomBox image to fix the random crashes I, and apparently you, have been suffering since the upgrade to Bullseye. I have heard that the normal updates have fixed the problem, but can’t verify that since I installed the latest weekly image and installed it today. Information from Johnny about update from Olimex
Just wanted to let you know.
Dave Oliver

hello,
I tried the 2022-01-07 img, and it lasted 1 day.
server crashed while using transmission.

I tried the 2022-01-07 img and it crashed three times over the last week. Still only running Cockpit, OpenVPN, and Samba. I did find 330 nearly identical log entries that occurred in rapid succession after the last two crashes:

" [1641941461.3557] dhcp4 (eth0): selecting lease failed: -131 NetworkManager" .

I am wondering if this could be some sort of external attack that renders the server unreachable. The router does indicate DOS attacks from various IP addresses. I have hardened my password and selected “Disable password authentication” under Secure Shell (SSH) Server in the System settings.

The first crash that occurred had something to do with either the automatic updates or the automatic backups which occur at night. I unscheduled automatic backups to see if that makes the FB more stable. I can report that the FB lasted through the night. I’ll report back if this solves the problem. The next thing to try is re-flash a new image and not select the recommended automatic updates.

Hello Johnny,
I believe my 10+ year old router/modem was letting DOS and DDOS attacks through and crashing my FBX within a day or two after rebooting or reinstalling newest FBX image. I replaced my modem with a new model and so-far-so-good; its been going for 3 whole days. Looking at the router/modem logs and FBX logs shows an increase in the attacks coming from around the world but I think most are being deflected. Only time will tell. If people are having crashes with old routers, it might be worthwhile to replace the insecure equipment. I will report back in a week or two. To see my experience with day one of the new router/modem go here https://discuss.freedombox.org/t/attacks-on-freedombox-from-around-the-world/1915

Hi all,
apologize for my late response. Unfortunately, The crashes remain so far, every 5 or 6 days my box does not respond anymore. The last time it went silent was today at 7:00. Here some lines from journalctl:

Feb 16 07:00:06 freedombox systemd[1]: Started Timeline of Snapper Snapshots.
Feb 16 07:00:06 freedombox systemd[1]: Started WordPress Scheduled Events Trigger (Cron).
Feb 16 07:00:06 freedombox dbus-daemon[366]: [system] Activating via systemd: service name='org.ope>
Feb 16 07:00:06 freedombox systemd[1]: Starting DBus interface for snapper...
Feb 16 07:00:06 freedombox dbus-daemon[366]: [system] Successfully activated service 'org.opensuse.>
Feb 16 07:00:06 freedombox systemd[1]: Started DBus interface for snapper.
Feb 16 07:00:06 freedombox systemd-helper[26412]: running timeline for 'root'.
-- Boot e7561208bb914a52b985bb8a66b9e17c --
Feb 16 08:10:50 freedombox kernel: Booting Linux on physical CPU 0x0
Feb 16 08:10:50 freedombox kernel: Linux version 5.10.0-11-armmp-lpae (debian-kernel@lists.debian.o>
Feb 16 08:10:50 freedombox kernel: CPU: ARMv7 Processor [410fc074] revision 4 (ARMv7), cr=30c5387d
Feb 16 08:10:50 freedombox kernel: CPU: div instructions available: patching division code
Feb 16 08:10:50 freedombox kernel: CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instructi>
Feb 16 08:10:50 freedombox kernel: OF: fdt: Machine model: Olimex A20-OLinuXino-LIME2
Feb 16 08:10:50 freedombox kernel: Memory policy: Data cache writealloc
Feb 16 08:10:50 freedombox kernel: efi: UEFI not found.
Feb 16 08:10:50 freedombox kernel: Reserved memory: created CMA memory pool at 0x000000004a000000, >
Feb 16 08:10:50 freedombox kernel: OF: reserved mem: initialized node default-pool, compatible id s>
Feb 16 08:10:50 freedombox kernel: Zone ranges:
Feb 16 08:10:50 freedombox kernel:   DMA      [mem 0x0000000040000000-0x000000006fffffff]
Feb 16 08:10:50 freedombox kernel:   Normal   empty
Feb 16 08:10:50 freedombox kernel:   HighMem  [mem 0x0000000070000000-0x000000007fffffff]
Feb 16 08:10:50 freedombox kernel: Movable zone start for each node
Feb 16 08:10:50 freedombox kernel: Early memory node ranges
Feb 16 08:10:50 freedombox kernel:   node   0: [mem 0x0000000040000000-0x000000007fffffff]
Feb 16 08:10:50 freedombox kernel: Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
Feb 16 08:10:50 freedombox kernel: On node 0 totalpages: 262144
Feb 16 08:10:50 freedombox kernel:   DMA zone: 1728 pages used for memmap
Feb 16 08:10:50 freedombox kernel:   DMA zone: 0 pages reserved
Feb 16 08:10:50 freedombox kernel:   DMA zone: 196608 pages, LIFO batch:63
Feb 16 08:10:50 freedombox kernel:   HighMem zone: 65536 pages, LIFO batch:15
Feb 16 08:10:50 freedombox kernel: psci: probing for conduit method from DT.
Feb 16 08:10:50 freedombox kernel: psci: Using PSCI v0.1 Function IDs from DT
Feb 16 08:10:50 freedombox kernel: percpu: Embedded 21 pages/cpu s54668 r8192 d23156 u86016
Feb 16 08:10:50 freedombox kernel: pcpu-alloc: s54668 r8192 d23156 u86016 alloc=21*4096
Feb 16 08:10:50 freedombox kernel: pcpu-alloc: [0] 0 [0] 1 
Feb 16 08:10:50 freedombox kernel: Built 1 zonelists, mobility grouping on.  Total pages: 260416
Feb 16 08:10:50 freedombox kernel: Kernel command line: console=ttyS0,115200 quiet
Feb 16 08:10:50 freedombox kernel: Dentry cache hash table entries: 131072 (order: 7, 524288 bytes,>
Feb 16 08:10:50 freedombox kernel: Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, l>
Feb 16 08:10:50 freedombox kernel: mem auto-init: stack:off, heap alloc:on, heap free:off
Feb 16 08:10:50 freedombox kernel: Memory: 896504K/1048576K available (12288K kernel code, 1680K rw>

At 8:10 I unplugged the power plug and reconnected a couple of seconds later. So the line before it went dumb reads:
freedombox systemd-helper[26412]: running timeline for 'root'

The next time my system goes down I’ll contact olimex myself.

Regards Oliver