I'm turning off SSH after a change of my opinion on ssh security (on linux)

I’ve advocated for using SSH here in the past. Today I’ve turned it off and think you should too. Two recent events lead me to believe that SSH is being attacked by a determined and resourceful adversary (one with an infinity budget, and I’ll leave you to guess who that would be).

  • RegreSSHion has reportedly been exploited by researchers and can be used to run arbitrary code.
  • xzUtils was nearly compromised which would have led to a compromise of SSH.

The xzUtils is an extremely sophisticated and deliberate attack that almost got distributed everywhere. This exploit leverages functions in xzUtils (data compression) which are then linked by systemd and this vulnerability is then passed through systemd to SSH. This was an extremely well thought out plan of military level sophistication. Systemd is garbage and I hate it - and now I know why. Everything that the init(1) crowd said about security is now proven 100% true.

RegreSSHion is similar in that foundational components used by SSH are compromised or exploitable. In this case it is glibc which is linked by basically everything. In case you aren’t aware of how serious this one is, legend has it that one of the original Unix developers, Ken Thompson, could use any Unix system because he compromised the C compiler used to build them. Who knows if this is true, but this is plausible if the exploit was done at the compiler level. glibc is only one step away from this.

It is important to point out that it is not the cryptography which is under attack - it is the foundational tools and programs that in both cases lead to SSH exploit. I see a pattern of organized attackers clawing at Linux, and by extension FreedomBox, system security however they can (read up on the compromise of /dev/random). This is exactly the way that Silk Road was compromised by the US government - by a foundational component of the site (Microsoft SQL Server if I recall).

OpenBSD, a competitor to Debian in the operating system space is not effected by either SSH exploit. Wouldn’t that be an awesome FreedomBox platform?

Recognize now that this is a pattern. Those who leave first leave best. I’ll let you know when I change my mind. I will be using Cockpit Terminal in place of SSH. I hope to post an update when I have confidence in SSH on FreedomBox.

3 Likes

Hi @joseph

I really appreciate what you have shared, though my knowledge is very limited and couldnt understand the technicality behind the threat.

In what you’re sharing, is the same exploit valid even if we use SSH keys to connect to our servers?

Thanks and best.

1 Like

I did not entirely read the full explanation but in the “Timing” part, there is

We do not wait until the last moment to send our (rather large) DSA
public-key packet to sshd

That seems to say that the attempt to exploit the vulnerability was done using public key authentication.

I am not an expert, but I would expect a web interface running a lot of scripts to be more at risk of security issues than a program like ssh.

This is possible but at least, ssh is open source software under scrutiny by many competent people, among which most if not all don’t want to allow such things.

Besides, many computers run proprietary firmware that is invisible to the OS (intel ME, raspberry pi boot firmware, etc) and if the adversary has power over US companies, it can simply ask them to provide access to these computers via the firmware, which can be nearly impossible to discover.

So anyone concerned with this should also make sure to only use computers that don’t need proprietary firmware to boot (like the Pioneer or the Rockpro64).

3 Likes

Systemd is garbage and I hate it - and now I know why. Everything that the init(1) crowd said about security is now proven 100% true.

Could not agree more, that’s why I started tinkering with Devuan a few weeks ago. Unfortunately Freedombox seems to be quite entangled with Systemd - at least via the Cockpit App (not sure about others) - and is not available for installation in Devuan.

1 Like

Rockpro64

Does the RockPro64 have open source firmware? Can you point me to a source for that info?

Yes, but let me clarify. The xzUtils exploit did not make it into the ecosystem - thankfully someone was able to detect this before it was rolled out. The RegreSSHion exploit exists in some cases - and I don’t know for FreedomBox today. So, what I see is a determined effort to compromise SSH on Linux and I’ve decided to take an SSH vacation. These two exploits do not attack the keys, do not attack the cryptography, or do not, “break the code.” I believe that they are both able to exploit SSH at the key-exchange stage, though.

From the command line you can look at any executable program with the ldd command and this will tell you all of the libraries that are linked when this program is compiled (created). The xzUtils exploit, for example, places the compromise in a data compression library which is then is linked, or included, in the program. So the exploit becomes available because the predefined library functions are insecure even if the encrypted data is very secure. This way the attack occurs when connecting by SSH - but somehow works around the key-exchange function to insert a program or data into the computer memory for execution. Here is ldd output for /bin/ls:

$ ldd /bin/ls
linux-vdso.so.1 (0x00007ffd2b179000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f5af2ecf000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5af2cee000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f5af2c54000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5af2f43000)

You’ll notice the libc.so.6 library. RegreSSHion uses this library as part of the exploit - but maybe not this particular version, or maybe we don’t have another tool that it used in the tool-chain to make the exploit. I just don’t know, but you’ll find libc linkage in many of your programs so I regard any whiff of a problem in libc to be pretty severe.

1 Like

I think for our purpose it is safe to think of this as a key spoof given that the exploit allows for execution of arbitrary code with root privilege. But it’s not exactly a key spoof. It is an exploit of the key-exchange dialog. I believe that there is some crafted communication here with a designed “malformed” packet(s). If you behave as though any private key is always compromised you’d do the right thing, so call it key spoofing.

I agree, but the big news is not about sophisticated attacks of this software stack. I can make it without SSH even if I don’t want to. Cockpit uses systemd, so yeah, it’s probably hosed already. SSH is a large population and a crafted attack of this would be very valuable. I think the Cockpit user base is a much smaller target less worthy of such effort by attackers.

We’re going to end up transferring files with uuencode and kermit as in the early 90’s at this rate.

But, isnt a library like libc so common in the ecosystem? i mean wouldnt a threat to such a library (libraries) have an immense effect on linux itself?

1 Like

So I’m not saying that is the case, but RegreSSHion boils down to something in libc. Whichever version and what else is needed - I don’t know and won’t be researching. I’m just going to avoid SSH exposure for a while.

1 Like

I’d be happy to read an update about how much of FreedomBox can work without systemd. Post about your Devuan adventure!

For the reader unfamiliar with Devuan, I believe that this is a Debian fork that does not include systemd.

What would be the motivation for this? Using a system that is less common so that it is less a target of attackers? Or is it because of the security reputation of OpenBSD?

With respect to the security aspect, OpenBSD web page warns that people working on OpenBSD do not have resources to audit packages as well as the base system. Would this well-audited base system still be beneficial from a security perspective when using all the additional packages required for services currently offered with Freedombox?

1 Like

OpenBSD has a sterling reputation for security and it is deserved. That said, I don’t think that migrating FreedomBox to OpenBSD is on the table and I’m not really advocating for that. I would choose it over Debian, though, I would only be worried about hardware support.

Open source code review is a powerful security assurance mechanism. I also believe that it is less-good than I thought as recently as a couple weeks ago. In my case I’d regard it as normalcy-bias to continue behaving as I did before. I’ll take a little break from running SSH while the ssh teams figure out the flaws in key-exchange.

I have to live on a mountain of dependencies to run FreedomBox and I accept this. I worry about Let’s Encrypt. Systemd is now a proven vulnerable attack surface. The state of SSL is reasonably up for debate. Yet, here I am. The ssh situation has crossed a line for me, and I’m doing something about it. I think it will probably fine, but this is a small sacrifice I can make for peace of mind.

There is a reasonable chance this is more of a media event, possibly even deliberately implanted, to cause people to not choose ssh. So maybe I play into the attacker’s hand. It’s a jungle out there. The good part is we all don’t have to do the same thing. You know what I’m doing and why. I’m avoiding the black-swan event which is most likely to never come, and I’ll be back to ssh in months.