Rsync vs. Samba+OpenVPN vs. WebDAV

Dear FreedomBox Team
You are my heroes! Thank you so much for your invaluable work on this project! You opened the window to self hosting and Linux for me. Through this I really gained freedom on my data and IT infrastructure. 1,5 year ago I left big tech and since then I was able to build a Linux IT infrastructure that is incredible and I wouldn’t believe it could be possible two years ago. I even have a FreedomBox backup server (rsync) in a remote location and this one is connected through WireGuard with my firewall and is backing up our data and email FreedomBoxes, so cool!

With love
David

3 Likes

By “Freedombox backup server”, do you mean another computer on which Freedombox is installed, or something else? You refer to rsync, did you add some backup using rsync manually to your Freedombox?

The backup app is using borg (perhaps borgmatic?), not rsync. It is nice but has the drawback that it does not include (samba) shares and syncthing data.

Hi @Avron

The following answer has mainly beginners in focus, which you aren’t, but maybe also answers your question.

Yes, the FreedomBox Borg backup is perfect! for doing a fresh install, move FreedomBox to a different machine or to fix a broken app.
With rsync I can also cover different use cases and copy almost the whole rootfs or specified parts of it like data etc. It can be done to a local storage device or to a remote computer. Rsync comes in to play, if you need to backup Samba or Syncthing, e.g. playlists from my Mopidy music server or whatever you need. There is another possibility for the need of a rsync backup: if it should be the case, that nothing goes anymore and you can’t reach plinth or you are not able to ssh in your FreedomBox anymore, then you can start your dd (or whatever) copy of your OS, even if it is many months old, and just play the latest rsync backup back to your FreedomBox, restart and you have a functioning OS again (it worked as I tested it). With rsync you can also make a remote backup of your local FreedomBox backup, whether it is stored on a local drive or in /var/lib/freedombox/borgbackup/. FreedomBox has a built in ability to remotely backup the FreedomBox. But if you choose to have a remote backup server for more than that, you can backup your fbx with rsync if needed and include the system given backup.

When I started with FreedomBox one and a half year ago, I missed a guide how to make backups from Samba and Syncthing, since I was an adopter of Windows before and didn’t had almost any clue about Linux and computers at all. Somebody reading this has still to work out details and knowledge about rsync etc. But with this an idea is given where to possibly evolve to in the case of simple backups.

Maybe I try to describe my setup, so somebody can use this or parts of it.

Since I run an email server with my family’s main address, I had to set up the whole infrastructure a bit over the minimum. Therefore I have a netgate pfsense+ firewall, which turned out to be a good choice for me. Pfsense+ allows the installation of the package pfblockerNG, which is highly customizable and keeps away most bad ips etc. I allmost do not have any entries on fail2ban on my FreedomBox (sometimes none for days). The firewall is also good, since I can set up WireGuard connections for all family members to reach Samba and Syncthing (this way I do not need to open SSH, Samba and Syncthing ports). And since the firewall does all the math for de- and encryption my Pioneers have more resources to do their task.

All 3 FreedomBoxes are Pioneers:
The hardware setup is on all 3 Pioneers the same: Pioneer with disk bay, where the SSD for data and local backups are located (all SSD’s with LUKS2 encrytion). The OS is on a 32 GB eMMc (RaspiKey) instead of a microSD. This makes it easier (compared to have the rootfs on a SSD) to sometimes dd copy the whole OS. And eMMc makes the storage device more reliable then microSD’s.

freedombox1 (local, static IP)
Data- and media server. Samba and Syncthing for data and Mopidy as music server.
All the Syncthing data is stored in Samba group_share, hence data is accessible over VPN on devices without Syncthing (e.g. a notebook for traveling where you don’t want data on it out of different reasons)

freedombox2 (local, static IP)
Emailserver, SOGo, Bepasty

freedombox3 (remote, DHCP, connectet to firewall/router at a good friends home)
Backupserver with rsync, WireGuard, Mailutils and Postfix (for email notification)

Aditionally to the FreedomBoxBackups I do local and remote backups with rsync, all with cronjobs as root.

RSYNC examples (I have more of them for different needs):

The backups are pulled (fbx3 remotely pulls the backups from fbx1 & fbx2).

In FreedomBox Plinth > System > SSH you find how to configure root pw and ssh connection.

# Backup fbx1 daily Samba (busamba)
15 10,12,15,18,22 * * * rsync -aW --partial --delete --exclude='xy' --exclude='xy' root@192.168.178.45:/media/samba/FreedomBox/shares/ /media/server3/fbx1/busamba
# Backup fbx1 boot daily (buboot)
25 21 * * * rsync -aW --delete root@192.168.178.45:/boot/ /media/server3/fbx1/buboot
# Backup fbx1 rootfs daily (bud)
30 21 * * * rsync -aAHWX --partial --delete --exclude=/.snapshots/* --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/run/* --exclude=/mnt/* --exclude=/media/* root@192.168.178.45:/ /media/server3/fbx1/bud

To recover a part or partition in the terminal of the remote backup server, you just need to change input output paths, for example (boot recovery, if the system is running):

# rsync -aW --delete /media/server3/fbx1/buboot/ root@192.168.178.45:/boot

Here is my script for incremental backups of samba with email notification:
The content of the log file gets automatically added to the email, so you do not have to open a file in your mail to display the output. The script is mainly compiled from these sources:
https://wiki.ubuntuusers.de/Skripte/Backup_mit_RSYNC/
https://tecadmin.net/backup-linux-system/

#!/bin/sh

source=root@192.168.178.45:/media/samba/FreedomBox/shares/
target=/media/server3/fbx1/samba_inkr/ 
date=$(date +%d-%m-%Y_%H:%M)

rm /home/rsynclog.txt

rsync -a --exclude='Pictures' --exclude='xy' --log-file=/home/rsynclog.txt "${source}" "${target}${date}/" --link-dest="${target}last/"
ln -nsf "${target}${date}" "${target}last"

if [ $? -eq 0 ]; then
    echo | mail -s "Backup Samba successful" xy@mydomain < /home/rsynclog.txt
else
    echo "Backup with error" | mail -s "Backup Samba error" xy@mydomain
fi

exit 0

With the --exclude= option you can recursively exclude folders, e.g. if you want your picture folder with all the subfolders excluded, just add the name of the folder between marks like in the above example.

Now, as I consider myself still as a beginner, there maybe better ways to do some of these steps. But this is how it works for me now, still learning.
I crashed my OS, when I created a cronjob with the wrong syntax on the options in rsync. The example option works in the terminal, but not as cronjob. Do not make a cron job with an option like this: --exclude={/.snapshots/,/dev/,/proc/,/sys/,/tmp/,/run/,/mnt/,/media/}
With this example I just want to say, that I have tried different ways and messed up sometimes (also experimented with the command line versions of backintime, timeshift and rsnapshot - but none was satisfying, rsync was straight forward).

Thanks for the details.

For rsync pulls, you had to install rsyncd, did you install rsyncd?

I avoid any customization of my machines running freedombox, in order to reduce the risk of breaking something there. As the backup feature of freedombox does not cover samba, I only need one samba share and it is easy to manually set this up without freedombox, I use another computer without freedombox for this. The most difficult was to setup borgmatic for backups to yet another computer.

For remote access:

  • I tried samba through openVPN on freedombox but it wasn’t working well, and it happened that openVPN that I had configured stopped working following upgrades on client or server, and different clients were not compatible with the same server options, which made it a complete headache.
  • I tried configuring wireguard with freedombox but never managed to make it work.
  • I now use sftp with restricted rights, to access the same data like the samba share. I wrote myself a guide to set this up.

In general, ssh, sftp and reverse ssh tunnels are my most commonly used tools. They are super stable and by learning a bit there is a massive amount of usages that become accessible.

I access my Samba shares via OpenVPN. It works well when you are not physically too far away, as in the same country.

I was abroad for several months this year and this method was practically unusable, because it is, unfortunately, too slow.

I was thinking about setting up WebDAV to see if I can interact with my shares remotely with hopefully better performance. But as Avron said, I also avoid changing too much in my FreedomBox to avoid breakages and downtime.

This awesome discussion deserves it own topic. Also the announcement thread should not get too many messages as many people only want to see the announcement messages and subscribe only to that category.

BTW, there is an open issue about adding WebDAV support to the sharing app. If you figure out the right Apache configuration changes, we can make this feature available to everyone.

Another thing from that discussion: Nextcloud also offers WebDAV support.

3 Likes

Thank you @Avron and @fefekrzr for your insights how you do it.

WireGuard leaves OpenVPN and IPsec in the dust, much faster…

@Avron
No, I just installed rsync: sudo apt install rsync
rsync won’t break anything and is no customization, since rsync is well implemented in Linux since almost 30 years and is the core of most major backup programs like backintime, timeshift, rsnapshot …
And it is easy to use, like the cp command.

Let’s say you just want to have your Samba share mirrored, you just need to run:

rsync -a --delete /var/lib/freedombox/shares/ /media/root/your_storage_device/folder

The --delete option removes the deleted files or folders in your source directory from the target directory.

Easily automate this with a cron job to do it daily or whatever.

1 Like

Is that already included in the Nextcloud installed by freedombox?

If webdav is added directly to Apache, it may become easy to have a samba share accessible with webdav, which would be really convenient.

1 Like

I use rsync on other machines, but even on a local network with rsync daemon (in order to save the processing due to ciphering by ssh), it takes a very long time. If I would do that on a machine running freedombox, I am a bit worried that a number of freedombox actions could be triggered during that time that could interfere with that.

Besides, rsync is convenient, but it won’t do deduplication and store multiple versions like borg(matic), that is used for the backup app, so I would really prefer that the backup app could be set to include (samba) share data.

1 Like

i too prefer to use borg (mostly due to its deduplication advantage). ive created simple bash scripts to take samba share backups. these scripts are then run by cron at specified times and intervals.

i have two local disks where i keep my backups: one is always connected to my fbx (utilizing cron), the other i connect once every month and store away, at my office. its not very tech-savy, i know : ) but does the job.

ive also setup fbx to store its own backups on the always online disk

As for the main discussion; im not sure webdav is a very secure option when compared with vpn+x. For ‘x’ i prefer sftp as its faster, and use smb for maybe small transfers. for vpn, i find wireguard much faster and prefer it over openvpn.

So, if i were going to take online backups, my route would probably be wireguard+nfs or wireguard+sshd with borg.

As I started with FreedomBox early 2024, I did try to figure out, how to use Borg of FreedomBox for backup my Samba shares and Syncthing content. But I couldn’t achieve it because my knowledge about Linux was not enough. Whith rsync I was able to do it. It seemed to be easier to understand (for me).
So it would be nice, to have a guide for beginners on how to do backups of Samba and Syncthing content with Borg. Maybe you should think about to write a little guide in tips and tricks?
The good thing about having a backup solution apart of FreedomBox backups is, that you can exclude parts you don’t need to backup, e.g big folders (like pictures) you manage somewhere else.

1 Like

A guide might be great.

But in the meantime, feel free to checkout the borg website, it has some very nice tutorials for beginners.

Addittionally, if you’re on linux theres a very nice flatpak app-frontend called Pika Backup. I use it a lot to look into existing archives, makes life easier : )

Once borg is sorted out vpn and secure connections can also be looked into.

Did you install Borg again or do you use Borg, that FreedomBox uses? Questions like this, is what I was looking for.
Avron mentioned Borgmatic and it also looks quite good as it is an “extended” Borg.
As soon as I have some time, I will take a look on these backup solutions.
Thanks @Ged and @Avron !

I configured samba and sftp on a computer without freedombox. I configured borg, with borgmatic, to backup on another computer, and it is run as a systemd user service daily.

I started writing a guide, the samba and sftp setup is complete, the samba and sftp client configuration for GNU/Linux, Windows, Mac, Android and iOS is written, but the borg(matic) part is still to be written, although I have some notes. Will be my next action. Also, I need to make an English version (French only now).

2 Likes

Borg is already in FBX. You dont have to install it (in fact, FBX uses Borg to make its own backups).
I have another server that i use borgmatic on. Cant say too much of a fan of it. If you’re not configuring different backups all the time, or if youre not constantly changing your configuration, a simple borg bash script works wonders.

1 Like