[SOLVED] SSH host public key could not be verified despite proper keys

[knightofnight]

Problem Description
Good day! I tried to make a remote backup location from my freedombox to my Ubuntu PC but it keeps giving me out the “SSH host public key could not be verified” error despite whatever I tried. I’m well aware that there have been past users like this and this that may have resolved their issues but it doesn’t seem to work for me.
I’ve tried

1. Clearing out known_hosts on /var/lib/plinth
2. Connecting to fbx via ssh on host PC
3. Connecting to host PC via ssh on fbx
4. Add public keys on known_hosts at /var/lib/plinth
5. Connecting to host PC via ssh on fbx’s cockpit’s admin and root

I’m pretty much stuck wondering what else to do.

Steps to Reproduce

1. Login to FreedomBox.
2. Go to Backup page.
3. Click on the Add Remote Backup Location Button.
4. Fill in the details
5. Verify the shown public key via ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub as was instructed
6. Confirm details and wait for SSH host to be verified.

Expected Results
I expected to see a message confirming that the SSH host is verified and also the public key, therefore the repository should be added and mounted as a backup location

Actual results
I get an error with the following message:

SSH host public key could not be verified. Repository removed.

Screenshot

Information

• FreedomBox version: 23.18
• Hardware: Raspberry Pi 4 Model B Rev 1.4
• How did you install FreedomBox?: ‘using a cloud image’

[Avron]

@knightofnight

• Clearing out known_hosts on /var/lib/plinth
• Connecting to fbx via ssh on host PC
• Connecting to host PC via ssh on fbx
• Add public keys on known_hosts at /var/lib/plinth
• Connecting to host PC via ssh on fbx’s cockpit’s admin and root

I am not sure I follow what you exactly did, perhaps you did too many things.

My suggestion:

• delete /var/lib/plinth/.ssh/known_hosts when logged via ssh to the freedombox
• run sudo journalctl -f in your terminal to see details
• try again adding the backup location in the web interface

If it still does not work, check on your terminal with the journal if there is not a suggestion to run an ssh-keygen command to solve the issue. When I get such a host key error when invoking ssh to a remote host from a terminal, I normally have this advice, I execute it, try ssh again and it works (or I get a different error).

[joseph]

Hi @knightofnight,

I was just able to make this work and there are a few steps. You’ll have to substitute your correct hostnames in these steps. We need to make sure that we have everything right with ssh from both sides and in both directions. I’ll refer to the backup target device as repohost.

Create SSH Keys for FreedomBox and Repohost users.

If you have not already done so:

• log into FreedomBox using Cockpit Terminal and run $ssh-keygen.
• log into the repohost and run $ssh-keygen.

Log into FreedomBox using Cockpit Terminal

Tell FreedomBox to trust repohost

$sudo nano /var/lib/plinth/.ssh/known_hosts
Use editor commands to get the repohost public keys into the nano edit buffer:

• make a new line at the bottom of the file
• Control-T to execute a command
• ssh-keyscan repohost.local
• remove the comment lines (starting with #) with Control-K
• Control-O to save the file

Tell the FreedomBox user to trust the repo host also

$nano ~/.ssh/known_hosts
Use editor commands to get the repohost public keys into the nano edit buffer:

• make a new line at the bottom of the file
• Control-T to execute a command
• $ssh-keyscan repohost.local
• remove the comment lines (starting with #) using Control-K
• Control-O to save the file

Tell the Freedombox user to trust the repohost user

$nano ~/.ssh/authorized_keys
Use editor commands to get the repohost user public keys into the edit buffer

• make a new line at the bottom of the file
• Control-T to execute a command
• $ssh repohost.local 'cat ~/.ssh/id_rsa.pub'
• Control-O to save

Now log in to repohost

Tell the repohost user to trust FreedomBox

$nano ~/.ssh/known_hosts

• make a new line at the bottom of the file
• Control-T to execute a command
• $ssh-keyscan freedombox.local
• remove the comment lines (starting with #) using Control-K
• Control-O to save the file

Tell the repohost user to trust the FreedomBox user

$nano ~/.ssh/authorized_keys
Use editor commands to get the freedombox user public keys into the edit buffer

• make a new line at the bottom of the file
• Control-T to execute a command
• $ssh freedombox.local 'cat ~/.ssh/id_rsa.pub'
• Control-O to save

What you’ve changed by all this

• FreedomBox trusts the repohost (/var/lib/plinth/.ssh/known_hosts)
• FreedomBox user trusts the repohost (~/.ssh/known_hosts)
• FreedomBox user trusts the repohost user (~/.ssh/authorized_keys)
• Repohost user trusts FreedomBox (~/.ssh/known_hosts)
• Repohost user trusts FreedomBox user (~/.ssh/authorized_keys)

Using these trusts all the right things can happen now:

• FreedomBox user makes a backup
• FreedomBox sets up an encrypted connection to repohost
• FreedomBox is sure that it is communicating with the repohost user
• Repohost allows the connection from the Freedombox
• Repohost is sure that it is communicating with the FreedomBox user
• FreedomBox sends all of your confidential information the correct user on the correct computer.
• Other people can’t read your backup as it crosses the network because it’s encrypted.
• Other people can’t fool your FreedomBox into sending your information to the wrong place because they don’t have the host and user keys needed to receive that connection.

Backup Encryption

Your backup will always be encrypted by SSH as it is transferred to the remote location. If your repohost folder can be accessed by other users who you would not want to allow access to the backup files (why would you do this?!) you can opt to encrypt the backup file. This will protect the information in the backup from someone with access to the file on the repohost disk.

Best Practice for Backup File Protection

I want to store my backup files in a place where people who don’t need to use backup files won’t be able to see or tamper with them.

Store the files in a folder owned by you. Remove filesystem permission for other users.
$mkdir backupFolder
$chmod 700 backupFolder
The backup software will do the right thing and not allow other users to see the contents of files or folders created by Backups.

-rw------- means only the user joseph can read and write the file.
drwx------ means only the user joseph can read, write, and list contents for the directory

I want have a backup repository on someone else’s computer…

This is a good case for backup file encryption to protect the backup contents while they are out of your complete control. I won’t discourage you from using the backup encryption on your own computer, either, but it is prudent when on someone else’s computer.

I use a USB disk to contain my backups.

Also encrypt, as these devices are easily lost or stolen.

[knightofnight]

Apologies for the late reply.

@Avron I tried sudo journalctl -f and also looked at it via Logs on the Cockpit terminal and it honestly didn’t really pop up something that seems worthwhile besides that it wasnt popping up any ssh-keygen suggestions.

FREEDOMBOX:443 192.168.50.1 - - [17/Jan/2024:00:23:40 +0800] "GET /plinth/sys/backups/repositories/add-remote/ HTTP/2.0" 200 4465 "https://FREEDOMBOX/plinth/sys/backups/repositories/add-remote/" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0"
FREEDOMBOX:443 192.168.50.1 - - [17/Jan/2024:00:23:38 +0800] "GET /plinth/sys/backups/repositories/9a93a4fc-b48b-11ee-b0ca-dca632cd0329/ssh-verify/ HTTP/2.0" 302 0 "https://FREEDOMBOX/plinth/sys/backups/repositories/add-remote/" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0"
pam_unix(sudo:session): session closed for user root
pam_unix(sudo:session): session opened for user root(uid=0) by (uid=110)
plinth : PWD=/ ; USER=root ; COMMAND=/usr/share/plinth/actions/actions backups is_mounted --write-fd 18
# backups..is_mounted(…)
/usr/bin/plinth
Connected (version 2.0, client OpenSSH_8.9p1)

@joseph Many thanks joseph for your elaborate answer, it was very clear with its intentions and easy to follow. I have to say that I’ve successfully made a connection and a backup with your answer as can be seen on the screenshot below (Although on a temporary test directory). I have to mention that for some reason doing PCNAME.local doesn’t really work for me (spouts out getaddrinfo errors and seems like it cant resolve it), and it only works if I address it via its IP.

Also I was trying to mount the backup location with a mounted external HDD on my PC which resides on /media/USER/HDDNAME but it threw me an [Errno 13] Permission Denied error. I’m trying to figure out a way to do this. Full Error log below as I can’t make another screenshot as a new user.
(EDIT) I’ve successfully mounted my external HDD as a backup location. The problem was that it was mounting using my current user, which I did not intend as I made a different user (fbxbackup) with its own ssh keys, and authenticated the entire process with that user in mind. The permission denied error was genuinely saying the permissions were denied.
The fix I did for this was to login to the other user and took ownership of the external hard drive via the Disks app on GNOME. Now it’s accurately showing up as /media/USER/Backup, and has mounted successfully.
(EDIT 2) That wasn’t a permanent change, a permanent change is changing the mount options of the drive (also using the Disks app) and making sure it’s mounted at the correct intended directory, mine is /media/USER/Backup. I think this is doable manually by fstab as the changes are applied there anyway, but doing it myself just crashed my PC, lol.

/usr/bin/plinth
Error adding repository: [Errno 13] Permission denied
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/plinth/modules/backups/views.py", line 425, in _save_repository
    repository.initialize()
  File "/usr/lib/python3/dist-packages/plinth/modules/backups/repository.py", line 430, in initialize
    self._ensure_remote_directory()
  File "/usr/lib/python3/dist-packages/plinth/modules/backups/repository.py", line 498, in _ensure_remote_directory
    sftp_client.listdir(dir_path)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 218, in listdir
    return [f.filename for f in self.listdir_attr(path)]
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 239, in listdir_attr
    t, msg = self._request(CMD_OPENDIR, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 822, in _request
    return self._read_response(num)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 874, in _read_response
    self._convert_status(msg)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 905, in _convert_status
    raise IOError(errno.EACCES, text)
PermissionError: [Errno 13] Permission denied

In hindsight while following the instructions, I guess the only steps that I’ve forgotten to do were to tell the users to trust the users (authorized_keys). I assume that’s the missing part, although I would assume freedombox should’ve done it automatically when it asked me to verify the shown public key. I find this behavior strange if it’s not achievable via the web and have to do this manually through console.

Thank you for the recommendations and practices for protection, before doing all this I made sure to make a separate user (fbxbackup) so that I’m certain my backups is only accessed by a user made for one purpose. Also I’ve ensured that it is encrypted. The only reason why I’m doing all this is because my Raspberry Pi couldn’t handle 2 hard drives so I opted to backup my freedombox through my PC instead.
Many thanks again for the answer, I really appreciated it.

[joseph]

@knightofnight

Many thanks joseph for your elaborate answer, it was very clear with its intentions and easy to follow.

I pleased to read this and thank you for that kind reply. I’m happy this is sorted for you.

[knightofnight]

I can’t seem to edit the topic title as [SOLVED] due to possibly being too new on the forums. Let it be known that my issue has been solved.

[Avron]

In my understanding:

• only the admin user can configure backups
• a remote location can be added using System->Backups->Add Remote Backup Location
• after entering someuser@somehost:~/path/, the password, encryption parameters
  • if this the first time the freedombox is connecting to somehost
    • the page asks to confirm the SHA256 hash of a host key of somehost
    • this will add the hash to /var/lib/plinth/.ssh/known_host
  • otherwise, plinth checks whether the host key of somehost matches with /var/lib/plinth/.ssh/known_host
    • if it does not match, plinth rejects the addition
• if the ssh connection with user@somehost is accepted with the provided password, the remove backup location is added

I tried this on a recently installed Pioneer on which I had not configured any backup location yet. At first, it failed, with the error indicated by @knightofnight.

Trying with an “older” Pioneer, I found out that:

• the name somehost wasn’t known, so I used the local IPv4 address instead
• connection was closed with “broken pipe”, which was because I used chroot for sftp but the chroot directory wasn’t owned and only writable by root (I have read that the group must be root too, but for me, it works even though it is not the case), so I fixed that.

Then it worked on the older Pioneer.

On the new pioneer, I deleted /var/lib/plinth/.ssh/known_host and tried again using the IP address of somehost and I selected the ed25519 host key instead of the RSA host key. It worked.

Why did it work before and not previous time? I don’t know.
Could the wrong ssh settings on somehost explain that? Perhaps.

@joseph

• FreedomBox user trusts the repohost (~/.ssh/known_hosts)
• FreedomBox user trusts the repohost user (~/.ssh/authorized_keys)
• Repohost user trusts FreedomBox (~/.ssh/known_hosts)
• Repohost user trusts FreedomBox user (~/.ssh/authorized_keys)

It seems to me that I don’t have any of this and it still works fine.