[SOLVED] Random errors, no SSH connection

Support
1

Problem Description

Tl;dr: I had random errors caused by “no space left on device” due to BTRFS snapshots.

Information

  • FreedomBox version: FB 21.4
  • Hardware: RaspberryPi 2 Model B
  • How did you install FreedomBox?: Official image on a 4G MicroSD card

Details

I was having random errors these recent days, mostly when trying to either access TinytinyRSS or connecting via SSH. TTRSS would not load and result in a HTTP 500 error page pointing me to the ‘status log’ - unfortunately the status log was not available since a login was not possible on the web UI using the provided URL. When navigating to the still running ‘/syncthing’ web UI I was able to get a valid user session and access the status log. It showed “no space left on device” errors.

I shut down the FB, removed the SD and plugged it into another computer. There I was able to delete old BTRFS snapshots and free up a little disk space. These two resource were really helpful:

Afterwards I put the SD card back into the Raspi and booted the FB again. All working fine.

As far as I could see the error started surfacing with a major apt upgrade happening. I have automatic updated enabled so it is quite possible that my FB will crash with the next run of unattended-upgraded the coming night.

I am currently considering buying a new SD card (the current one is only 4G) and re-installing my FB using one of the backups I made. I’ll update this post if I encounter any news on this. In my snapshots settings I have set the “keep free” value to 40%, hoping that this helps mitigate this issue.

I hope this helps somebody else. I love my FB and am very happy with the overall experience over the past few years. Thanks a lot to everybody who contributes to making this possible.

Cheers,
Axel

2

I have “Software Installation Snapshots” set to disabled on my box, because I found it created so many snapshots, and the timeline snapshots were sufficient for my needs.

1 Like
3

Thanks for the heads up, @jvalleroy. I’ll check my settings for snapshots, too - up to now I simply went with the defaults.

After the nightly upgrade run it crashed again yesterday. I have already found a new, bigger SD card and will migrate the partitions and data over, probably next weekend. Resizing a BTRFS filesystem seems to be straightforward to me:

4

Said and done, here’s what worked for me:

  • Power down my Raspi, remove the SD card (4G)
  • Create an image from the data on it on my desktop using a card reader:
    sudo dd if=/dev/mmcblk0 of=sdcard.img bs=4M status=progress
  • Swap the SD card in my reader for a bigger one (128G)
  • First remove all partitions from it (I don’t think this is necessary when dd’ing data onto it later, but I’m a thorough person):
    sudo cfdisk /dev/mmcblk0
    # remove all existing partitions, write & quit
  • Then, dd the image onto the new card:
    sudo dd if=sdcard.img of=/dev/mmcblk0 bs=4M status=progress oflag=sync
  • Afterward resize the partition that was originally too small:
    sudo cfdisk /dev/mmcblk0
    #select partition, choose 'resize' to max, write & quit
  • Then resize the filesystem to use all of the partition:
    sudo btrfs filesystem resize max /media/kopfkind/<mountpoint>

Afterwards I simply put the new, big SD card back into the Pi and booted it. Works like a charm!

Cheers & HTH,
Axel