Backup failure (again)

I tried a manual backup, it failed nearly immediately (well, after taking time stopping services).

Here are the logs:

Nov 09 10:35:43 fbox /usr/bin/plinth[532]: Running backup_pre hooks
Nov 09 10:35:43 fbox /usr/bin/plinth[532]: # backups..create_archive(…) 
Nov 09 10:35:44 fbox sudo[11861]:   plinth : PWD=/ ; USER=root ; COMMAND=/usr/share/plinth/actions/actions backups create_archive --write-fd 18
Nov 09 10:35:44 fbox sudo[11861]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=110)
Nov 09 10:35:53 fbox /usr/share/plinth/actions/actions[11863]: Error executing action: Command '['borg', 'create', '--json', '/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132', '/etc/letsencrypt/', '/etc/coturn', '/var/lib/ejabberd/', '/etc/ejabberd/letsencrypt/', '/etc/openvpn/', '/var/lib/quassel/', '/etc/radicale/', '/var/lib/radicale/', '/etc/postfix/', '/etc/dovecot/conf.d/', '/etc/rspamd/', '/var/lib/postfix/freedombox-aliases/', '/var/lib/rspamd/', '/var/mail/', '/etc/timezone', '/etc/snapper/configs/root', '/etc/default/snapper', '/etc/ssh/ssh_host_ecdsa_key', '/etc/ssh/ssh_host_ecdsa_key.pub', '/etc/ssh/ssh_host_ed25519_key', '/etc/ssh/ssh_host_ed25519_key.pub', '/etc/ssh/ssh_host_rsa_key', '/etc/ssh/ssh_host_rsa_key.pub', '/etc/apt/apt.conf.d/20auto-upgrades', '/etc/popularity-contest.d/freedombox.conf', '/etc/ejabberd/ejabberd.yml', '/etc/ejabberd/ejabberd.pem', '/etc/aliases', '/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json']' returned non-zero exit status 2.
                                                               Traceback (most recent call last):
                                                                 File "/usr/share/plinth/actions/actions", line 92, in _call
                                                                   return_values = func(*arguments['args'], **arguments['kwargs'])
                                                                 File "/usr/lib/python3/dist-packages/plinth/modules/backups/privileged.py", line 168, in create_archive
                                                                   _run(command, encryption_passphrase)
                                                                 File "/usr/lib/python3/dist-packages/plinth/modules/backups/privileged.py", line 350, in _run
                                                                   return subprocess.run(cmd, check=check, env=env, **kwargs)
                                                                 File "/usr/lib/python3.9/subprocess.py", line 528, in run
                                                                   raise CalledProcessError(retcode, process.args,
                                                               subprocess.CalledProcessError: Command '['borg', 'create', '--json', '/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132', '/etc/letsencrypt/', '/etc/coturn', '/var/lib/ejabberd/', '/etc/ejabberd/letsencrypt/', '/etc/openvpn/', '/var/lib/quassel/', '/etc/radicale/', '/var/lib/radicale/', '/etc/postfix/', '/etc/dovecot/conf.d/', '/etc/rspamd/', '/var/lib/postfix/freedombox-aliases/', '/var/lib/rspamd/', '/var/mail/', '/etc/timezone', '/etc/snapper/configs/root', '/etc/default/snapper', '/etc/ssh/ssh_host_ecdsa_key', '/etc/ssh/ssh_host_ecdsa_key.pub', '/etc/ssh/ssh_host_ed25519_key', '/etc/ssh/ssh_host_ed25519_key.pub', '/etc/ssh/ssh_host_rsa_key', '/etc/ssh/ssh_host_rsa_key.pub', '/etc/apt/apt.conf.d/20auto-upgrades', '/etc/popularity-contest.d/freedombox.conf', '/etc/ejabberd/ejabberd.yml', '/etc/ejabberd/ejabberd.pem', '/etc/aliases', '/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json']' returned non-zero exit status 2.
Nov 09 10:35:54 fbox sudo[11861]: pam_unix(sudo:session): session closed for user root
Nov 09 10:35:54 fbox /usr/bin/plinth[532]: Error running action backups..create_archive(*('/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132', ['/etc/letsencrypt/', '/etc/coturn', '/var/lib/ejabberd/', '/etc/ejabberd/letsencrypt/', '/etc/openvpn/', '/var/lib/quassel/', '/etc/radicale/', '/var/lib/radicale/', '/etc/postfix/', '/etc/dovecot/conf.d/', '/etc/rspamd/', '/var/lib/postfix/freedombox-aliases/', '/var/lib/rspamd/', '/var/mail/', '/etc/timezone', '/etc/security/access.d/50freedombox.conf', '/etc/snapper/configs/root', '/etc/default/snapper', '/etc/ssh/ssh_host_ecdsa_key', '/etc/ssh/ssh_host_ecdsa_key.pub', '/etc/ssh/ssh_host_ed25519_key', '/etc/ssh/ssh_host_ed25519_key.pub', '/etc/ssh/ssh_host_rsa_key', '/etc/ssh/ssh_host_rsa_key.pub', '/etc/apt/apt.conf.d/20auto-upgrades', '/etc/popularity-contest.d/freedombox.conf', '/etc/ejabberd/ejabberd.yml', '/etc/ejabberd/ejabberd.pem', '/etc/aliases', '/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json']), **{'comment': None, 'encryption_passphrase': None}): Command '['borg', 'create', '--json', '/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132', '/etc/letsencrypt/', '/etc/coturn', '/var/lib/ejabberd/', '/etc/ejabberd/letsencrypt/', '/etc/openvpn/', '/var/lib/quassel/', '/etc/radicale/', '/var/lib/radicale/', '/etc/postfix/', '/etc/dovecot/conf.d/', '/etc/rspamd/', '/var/lib/postfix/freedombox-aliases/', '/var/lib/rspamd/', '/var/mail/', '/etc/timezone', '/etc/snapper/configs/root', '/etc/default/snapper', '/etc/ssh/ssh_host_ecdsa_key', '/etc/ssh/ssh_host_ecdsa_key.pub', '/etc/ssh/ssh_host_ed25519_key', '/etc/ssh/ssh_host_ed25519_key.pub', '/etc/ssh/ssh_host_rsa_key', '/etc/ssh/ssh_host_rsa_key.pub', '/etc/apt/apt.conf.d/20auto-upgrades', '/etc/popularity-contest.d/freedombox.conf', '/etc/ejabberd/ejabberd.yml', '/etc/ejabberd/ejabberd.pem', '/etc/aliases', '/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json']' returned non-zero exit status 2. (2, ['borg', 'create', '--json', '/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132', '/etc/letsencrypt/', '/etc/coturn', '/var/lib/ejabberd/', '/etc/ejabberd/letsencrypt/', '/etc/openvpn/', '/var/lib/quassel/', '/etc/radicale/', '/var/lib/radicale/', '/etc/postfix/', '/etc/dovecot/conf.d/', '/etc/rspamd/', '/var/lib/postfix/freedombox-aliases/', '/var/lib/rspamd/', '/var/mail/', '/etc/timezone', '/etc/snapper/configs/root', '/etc/default/snapper', '/etc/ssh/ssh_host_ecdsa_key', '/etc/ssh/ssh_host_ecdsa_key.pub', '/etc/ssh/ssh_host_ed25519_key', '/etc/ssh/ssh_host_ed25519_key.pub', '/etc/ssh/ssh_host_rsa_key', '/etc/ssh/ssh_host_rsa_key.pub', '/etc/apt/apt.conf.d/20auto-upgrades', '/etc/popularity-contest.d/freedombox.conf', '/etc/ejabberd/ejabberd.yml', '/etc/ejabberd/ejabberd.pem', '/etc/aliases', '/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json'], b'', b'Killed stale lock fbox.metani.eu@2242048483580.18261-0.\nRemoved stale exclusive roster lock for host fbox.metani.eu@2242048483580 pid 18261 thread 0.\nRemoved stale exclusive roster lock for host fbox.metani.eu@2242048483580 pid 18261 thread 0.\nLocal Exception\nTraceback (most recent call last):\n  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4690, in main\n    exit_code = archiver.run(args)\n  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4622, in run\n    return set_ec(func(args))\n  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 177, in wrapper\n    return method(self, args, repository=repository, **kwargs)\n  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 586, in do_create\n    with Cache(repository, key, manifest, do_files=args.cache_files, progress=args.progress,\n  File "/usr/lib/python3/dist-packages/borg/cache.py", line 387, in __new__\n    return local()\n  File "/usr/lib/python3/dist-packages/borg/cache.py", line 379, in local\n    return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync,\n  File "/usr/lib/python3/dist-packages/borg/cache.py", line 464, in __init__\n    self.open()\n  File "/usr/lib/python3/dist-packages/borg/cache.py", line 512, in open\n    self.rollback()\n  File "/usr/lib/python3/dist-packages/borg/cache.py", line 622, in rollback\n    shutil.copy(os.path.join(txn_dir, \'config\'), self.path)\n  File "/usr/lib/python3.9/shutil.py", line 418, in copy\n    copyfile(src, dst, follow_symlinks=follow_symlinks)\n  File "/usr/lib/python3.9/shutil.py", line 264, in copyfile\n    with open(src, \'rb\') as fsrc, open(dst, \'wb\') as fdst:\nFileNotFoundError: [Errno 2] No such file or directory: \'/root/.cache/borg/b90624fb3aa8831282be9f75224e6947396a86df04c1665ce8c95b6136ada434/txn.active/config\'\n\nPlatform: Linux fbox 5.10.0-19-armmp-lpae #1 SMP Debian 5.10.149-2 (2022-10-21) armv7l\nLinux: Unknown Linux  \nBorg: 1.1.16  Python: CPython 3.9.2 msgpack: 0.5.6.+borg1\nPID: 11865  CWD: /\nsys.argv: [\'/usr/bin/borg\', \'create\', \'--json\', \'/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132\', \'/etc/letsencrypt/\', \'/etc/coturn\', \'/var/lib/ejabberd/\', \'/etc/ejabberd/letsencrypt/\', \'/etc/openvpn/\', \'/var/lib/quassel/\', \'/etc/radicale/\', \'/var/lib/radicale/\', \'/etc/postfix/\', \'/etc/dovecot/conf.d/\', \'/etc/rspamd/\', \'/var/lib/postfix/freedombox-aliases/\', \'/var/lib/rspamd/\', \'/var/mail/\', \'/etc/timezone\', \'/etc/snapper/configs/root\', \'/etc/default/snapper\', \'/etc/ssh/ssh_host_ecdsa_key\', \'/etc/ssh/ssh_host_ecdsa_key.pub\', \'/etc/ssh/ssh_host_ed25519_key\', \'/etc/ssh/ssh_host_ed25519_key.pub\', \'/etc/ssh/ssh_host_rsa_key\', \'/etc/ssh/ssh_host_rsa_key.pub\', \'/etc/apt/apt.conf.d/20auto-upgrades\', \'/etc/popularity-contest.d/freedombox.conf\', \'/etc/ejabberd/ejabberd.yml\', \'/etc/ejabberd/ejabberd.pem\', \'/etc/aliases\', \'/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json\']\nSSH_ORIGINAL_COMMAND: None\n\n\x1b[31m   ERROR\x1b[0m \x1b[94m__main__            \x1b[0m Error executing action: Command \'[\'borg\', \'create\', \'--json\', \'/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132\', \'/etc/letsencrypt/\', \'/etc/coturn\', \'/var/lib/ejabberd/\', \'/etc/ejabberd/letsencrypt/\', \'/etc/openvpn/\', \'/var/lib/quassel/\', \'/etc/radicale/\', \'/var/lib/radicale/\', \'/etc/postfix/\', \'/etc/dovecot/conf.d/\', \'/etc/rspamd/\', \'/var/lib/postfix/freedombox-aliases/\', \'/var/lib/rspamd/\', \'/var/mail/\', \'/etc/timezone\', \'/etc/snapper/configs/root\', \'/etc/default/snapper\', \'/etc/ssh/ssh_host_ecdsa_key\', \'/etc/ssh/ssh_host_ecdsa_key.pub\', \'/etc/ssh/ssh_host_ed25519_key\', \'/etc/ssh/ssh_host_ed25519_key.pub\', \'/etc/ssh/ssh_host_rsa_key\', \'/etc/ssh/ssh_host_rsa_key.pub\', \'/etc/apt/apt.conf.d/20auto-upgrades\', \'/etc/popularity-contest.d/freedombox.conf\', \'/etc/ejabberd/ejabberd.yml\', \'/etc/ejabberd/ejabberd.pem\', \'/etc/aliases\', \'/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json\']\' returned non-zero exit status 2.\nTraceback (most recent call last):\n  File "/usr/share/plinth/actions/actions", line 92, in _call\n    return_values = func(*arguments[\'args\'], **arguments[\'kwargs\'])\n  File "/usr/lib/python3/dist-packages/plinth/modules/backups/privileged.py", line 168, in create_archive\n    _run(command, encryption_passphrase)\n  File "/usr/lib/python3/dist-packages/plinth/modules/backups/privileged.py", line 350, in _run\n    return subprocess.run(cmd, check=check, env=env, **kwargs)\n  File "/usr/lib/python3.9/subprocess.py", line 528, in run\n    raise CalledProcessError(retcode, process.args,\nsubprocess.CalledProcessError: Command \'[\'borg\', \'create\', \'--json\', \'/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132\', \'/etc/letsencrypt/\', \'/etc/coturn\', \'/var/lib/ejabberd/\', \'/etc/ejabberd/letsencrypt/\', \'/etc/openvpn/\', \'/var/lib/quassel/\', \'/etc/radicale/\', \'/var/lib/radicale/\', \'/etc/postfix/\', \'/etc/dovecot/conf.d/\', \'/etc/rspamd/\', \'/var/lib/postfix/freedombox-aliases/\', \'/var/lib/rspamd/\', \'/var/mail/\', \'/etc/timezone\', \'/etc/snapper/configs/root\', \'/etc/default/snapper\', \'/etc/ssh/ssh_host_ecdsa_key\', \'/etc/ssh/ssh_host_ecdsa_key.pub\', \'/etc/ssh/ssh_host_ed25519_key\', \'/etc/ssh/ssh_host_ed25519_key.pub\', \'/etc/ssh/ssh_host_rsa_key\', \'/etc/ssh/ssh_host_rsa_key.pub\', \'/etc/apt/apt.conf.d/20auto-upgrades\', \'/etc/popularity-contest.d/freedombox.conf\', \'/etc/ejabberd/ejabberd.yml\', \'/etc/ejabberd/ejabberd.pem\', \'/etc/aliases\', \'/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json\']\' returned non-zero exit status 2.\n') ['  File "/usr/share/plinth/actions/actions", line 92, in _call\n    return_values = func(*arguments[\'args\'], **arguments[\'kwargs\'])\n', '  File "/usr/lib/python3/dist-packages/plinth/modules/backups/privileged.py", line 168, in create_archive\n    _run(command, encryption_passphrase)\n', '  File "/usr/lib/python3/dist-packages/plinth/modules/backups/privileged.py", line 350, in _run\n    return subprocess.run(cmd, check=check, env=env, **kwargs)\n', '  File "/usr/lib/python3.9/subprocess.py", line 528, in run\n    raise CalledProcessError(retcode, process.args,\n']
Nov 09 10:35:54 fbox /usr/bin/plinth[532]: Internal Server Error: /plinth/sys/backups/create/
                                           Traceback (most recent call last):
                                             File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 34, in inner
                                               response = get_response(request)
                                             File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 115, in _get_response
                                               response = self.process_exception_by_middleware(e, request)
                                             File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 113, in _get_response
                                               response = wrapped_callback(request, *callback_args, **callback_kwargs)
                                             File "/usr/lib/python3/dist-packages/django/views/generic/base.py", line 71, in view
                                               return self.dispatch(request, *args, **kwargs)
                                             File "/usr/lib/python3/dist-packages/django/views/generic/base.py", line 97, in dispatch
                                               return handler(request, *args, **kwargs)
                                             File "/usr/lib/python3/dist-packages/django/views/generic/edit.py", line 142, in post
                                               return self.form_valid(form)
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/views.py", line 128, in form_valid
                                               repository.create_archive(name, selected_apps)
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/repository.py", line 183, in create_archive
                                               api.backup_apps(_backup_handler, path=archive_path, app_ids=app_ids,
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/api.py", line 131, in backup_apps
                                               _run_operation(backup_handler, packet,
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/api.py", line 423, in _run_operation
                                               handler(packet, encryption_passphrase=encryption_passphrase)
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/__init__.py", line 96, in _backup_handler
                                               privileged.create_archive(packet.path, paths,
                                             File "/usr/lib/python3/dist-packages/plinth/actions.py", line 57, in wrapper
                                               return _run_privileged_method_as_process(module_name, action_name,
                                             File "/usr/lib/python3/dist-packages/plinth/actions.py", line 116, in _run_privileged_method_as_process
                                               return _wait_for_return(*wait_args)
                                             File "/usr/lib/python3/dist-packages/plinth/actions.py", line 154, in _wait_for_return
                                               raise exception
                                           subprocess.CalledProcessError: Command '['borg', 'create', '--json', '/media/bda056a4-4a23-11ec-a475-020a0480e0fc::Manuel-2022-11-09-1132', '/etc/letsencrypt/', '/etc/coturn', '/var/lib/ejabberd/', '/etc/ejabberd/letsencrypt/', '/etc/openvpn/', '/var/lib/quassel/', '/etc/radicale/', '/var/lib/radicale/', '/etc/postfix/', '/etc/dovecot/conf.d/', '/etc/rspamd/', '/var/lib/postfix/freedombox-aliases/', '/var/lib/rspamd/', '/var/mail/', '/etc/timezone', '/etc/snapper/configs/root', '/etc/default/snapper', '/etc/ssh/ssh_host_ecdsa_key', '/etc/ssh/ssh_host_ecdsa_key.pub', '/etc/ssh/ssh_host_ed25519_key', '/etc/ssh/ssh_host_ed25519_key.pub', '/etc/ssh/ssh_host_rsa_key', '/etc/ssh/ssh_host_rsa_key.pub', '/etc/apt/apt.conf.d/20auto-upgrades', '/etc/popularity-contest.d/freedombox.conf', '/etc/ejabberd/ejabberd.yml', '/etc/ejabberd/ejabberd.pem', '/etc/aliases', '/var/lib/plinth/backups-manifests/mediabda056a4-4a23-11ec-a475-020a0480e0fcManuel-2022-11-09-1132.json']' returned non-zero exit status 2.
Nov 09 10:35:54 fbox apache-access[1030]: metani.eu:443 192.168.1.51 - - [09/Nov/2022:10:32:29 +0000] "POST /plinth/sys/backups/create/ HTTP/2.0" 500 6616 "https://metani.eu/plinth/sys/backups/create/" "Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0"

After that, diagnostics show many services are not working, e.g. ejabberd, coturn, postfix. I saved logs ahead and after if that is usefull (it is to long to post here). I can make a bug report if useful.

Relevant bits from the error log:

Killed stale lock fbox.metani.eu@2242048483580.18261-0.
Removed stale exclusive roster lock for host fbox.metani.eu@2242048483580 pid 18261 thread 0.
Removed stale exclusive roster lock for host fbox.metani.eu@2242048483580 pid 18261 thread 0.
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4690, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4622, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 586, in do_create
    with Cache(repository, key, manifest, do_files=args.cache_files, progress=args.progress,
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 387, in __new__
    return local()
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 379, in local
    return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync,
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 464, in __init__
    self.open()
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 512, in open
    self.rollback()
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 622, in rollback
    shutil.copy(os.path.join(txn_dir, 'config'), self.path)
  File "/usr/lib/python3.9/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.9/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/borg/b90624fb3aa8831282be9f75224e6947396a86df04c1665ce8c95b6136ada434/txn.active/config'

Do you see any disk or filesystem errors in journal or dmesg? Could you try again after removing /root/.cache/borg folder?

No but I rebooted after the failure in order to re-start all services.

It is a remote backup, not local disk. On the remote machine, I have set “ForceCommand internal-sftp” for the user. It was working previously. When logged in as admin user to the freedombox, sftp in command line to the remote machine works.

Sure, doing that in a short moment.

EDIT: I tried setting a backup location on another remote machine, it fails with a message saying “Authentiucation public key of the server could not be verified”. Is it something in “known_hosts”? Of which user?

When a new backup location is added, we show in the interface the public key fingerprint of the server and ask user to verify it. Try removing the remote location and adding a new location instead of editing the location.

It worked ! Many thanks ! (this is a huge relief for me).

When trying to configure another remote machine for backup, I realized I can’t remember whether I have set encryption for backups and I can’t find anything for that in my password database.

I need to look at that.

I did but it says (my own translation to English):

Add a new remote ssh repository (green background)
SSH server already verified (green background)
Authentication public key of the server could not be verified (red background)
Repository removed (red background).

I think I re-installed the OS on the remote machine and did not keep the host keys, so probably the freedombox has stored the host key for it somewhere and it does not match with the new host key. I looked in /root/.ssh, there is nothing there. Not sure where to look.

Edit the file /var/lib/plinth/.ssh/known_hosts