Erlang Elixir Node Security Flaws

This is an ATTENTION to all Elixir/Phoenix/Erlang users out there that use distributed nodes or software that uses nodes. Whenever I talk about Erlang or Elixir you can interpret both since it's the same underlying system at work.
I'm not talking about node.js. But about the distributed OTP programming system that beats the hell out every other programming language.
All programming languages want to do distributed programming like Elixir and Erlang do when they grow up. I've recently started implementing distributed OTP systems and I'm still awed by the powers of the Elixir, Erlang and OTP ecosystem!
NOTE: Windows users... you might have to figure out how to do it with Putty:) There is an inherent security "flaw" which could lead to big problems if let unattended. Read below for more info

In November 2017 I discovered Elixir which is based upon Erlang. I fell in love, and from the first moment, I read the Elixir Getting Started guide (.EPUB file download) I felt something mysterious.
Reading the code even though it was for the first time AND it's a functional language made it feel right.
I've never felt this before with any other language except Tcl which I grew up with. You can read more about it in its own special article.

Erlang and Elixir is a great ecosystem. I've been told that before and I didn't believe it until i started reading code and testing things out myself.
Before I wanted to start out creating production software I wanted to test out its security potential.
Compared to other programming languages and tools it is very robust.
You probably won't find overflow problems, stack smashing is impossible since everything is held in the HEAP and the stack is simulated.
Command injection *is* possible but that depends on the app itself.

One other issue I've detected is the usage of :erlang.term_to_binary and :erlang.binary_to_term. This is a way to serialize data to binary when sending it over the wire or saving it in files.
It's an alternative to XML, JSON etc. It can be used for blockchain encryption and stuff.
Since internally this uses zlib compression and it's used quite a lot.. it can bring problems like binary zip bombs.
Why? Well, everything is stored in memory. And when decompressing a binary bomb that is only 4 MB you can use up to 4 GB of memory.

But that's not the problem of the day.

Erlang Node Seurity

Erlang and Elixir's distributed node system has another inherent weakness. It's cookie based.

It's advised that you don't run Erlang on internet facing websites.. But with the rise of Elixir and Phoenix this becomes quite complicated.

One of the first problems is that by default the Distributed Node system works in plaintext.
Another problem is that node authentication is cookie based which is deterministically generated.
All nodes share the same cookie. So if a node is compromised, all your systems can compromised. Now, meaning that the attacker knows these things and is an Erlang fan. But we don't want to take any chances, do we?

Compromised how?

Well, you can run code on the operating system via the :os.cmd() function. You can open a remote shell from within Erlang/Elixir itself. Another way is that you can actually compile code across the distribution via the 1 node.

One security researcher pointed out that you can actually guess a cookie because of the way atoms work in Erlang.

You can view a full talk about the issue here: https://www.youtube.com/watch?v=42k70Y-yTYY
So basically atoms are limited in Erlang (and elixir) he claims to have tested this and that he can guess the cookie atom based on a connection he can make using a node name
This is interesting in 2 ways:
You could guess the cookie by depleting the atom resources
You can CRASH a VM if it tries to recreate all the atoms

Both these things are potentially a problem

Another problem is the usage of the elixir iex --remsh or remote shell option.
Because the way of how the distributed node system works as explained earlier you can have problems on your OWN pc when debugging.
If you connect with your development computer to a compromised node via --remsh you become part of the node, so in theory and practice
the node you've connected to CAN run code on YOUR PC.
This means that your private files and even SSH keys can be read.

Ouch, this hurts!

Potential Solutions

Don't use Distributed Nodes

Well, if you only have 1 system.. You could just.. not use the distributed node system and you're safe.
This means that you should never use the Elixir --sname or --name function unless you are willing to risk things..
But Elixir and Erlang is BUILT for distribution and everything is internet connected today. This means that any startup probably runs their VM's in the cloud and needs to interconnect them.

Ok, so the next logical thing would be to implement something similar in Elixir and use it.
This is not such an easy task since it would mean to reimplement the wheel.. And it would probably NOT be as useful as the node issue.

If you don't need the power of distributed nodes then don't use them.
If you are willing to write your own distribution mechanism then you can implement security in any way you want.
As mentioned earlier, maybe it's not a good idea to reimplement the wheel.

However... even the simple usage of :observer.start REQUIRES the usage of the cookie-based node system.
So when you try to "debug" your Phoenix app.. Heh.. Guess what, you've opened up the door to the unknown.
But let's be serious, why use Elixir or Erlang if you are not thinking about using it's full potential and aren't into distributed computing?

Setting up Erlang to use TLS/SSL

A solution to the problem can be to setup Erlang to be TLS/SSL based. Yaay!
This can be done, it will be a tedious setup since you have to setup certificates with public key encryption..
Be sure that all your nodes are up to date.
But you then still need to secure epmd which will still continue to talk UNENCRYPTED.

Securing your certificates and server is ALSO important. This, however, CAN and SHOULD be automated.

Using SSH

This solution will solve the problems explained earlier except for the term_to_binary issues which are up to the developers to secure.

My recommendation is to setup a firewall that blocks everything. Setup SELINUX if required.

Then, when starting up any Erlang or Elixir based apps
Define the full name --name yourname@127.0.0.1
BE SURE to put the machine name of the node as 127.0.0.1 (0r 127.0.2.4 or whatever in the 127.0.0.1/8 range) FOR ALL your NODES.
This will make sure that when someone tries to connect to it.. they will be able only if they're local hosted. I know you might find this  slightly backward but read on.

Whenever a node needs to connect to another node setup a SSH reverse port connection to an low privileged user.
ssh -L4369:localhost:4369 -L39974:localhost:39974 edge@node2.host.example

This sets up a local port on your machine on localhost which then connects to the server and tunnels everything directly. The 4369 port is for the EPDM service on the remote server. You will probably have to setup multiple and different local ports for it since each node has it's own epdm service on 4369. SSH easily supports this.

Then connect via the local port. This will have 2 benefits.
First, you'll have encrypted traffic. Second your ports will ONLY be available on the nodes themselves. Two birds one stone. The only downside is that you'll have to write some scripts to automate the SSH port forwarding. Also take care to implement authentication via public key systems and don't pass passwords around. This will involve a little bit of overhead but from a security standpoint it's worth it.

NOTE: if you want to use the observer functionality you will need Xforwarding

# ~/.ssh/configHost ci  Hostname 192.168.1.100  User username  ForwardX11 yes  ForwardX11Trusted yes

Then whenever you connect to the node, you do it via the local FQDN. Even for debugginv via --remsh you first open a ssh -L connection to localhost AND connect to the localhost port.
The setup of this process can be automated.

Oh and one more thing. If you ARE using Nodes with cookies
Read more on the erlang:set_cookie/2 and/or Node.set_cookie . You can set different cookies for each node.
This will minimize the problem if one node is compromised since the attacker won't be able to connect to the other ones

At least untill he figures a way to get the cookies of the other nodes.

To sum it up

Don't use --sname --remsh --name on public facing computers UNLESS you've setup your SSH, firewall and other security systems.
Have different cookies
Use ssh for security

Resources to help you understand more

You can view a full talk about the issue here: https://www.youtube.com/watch?v=42k70Y-yTYY
https://mfeckie.github.io/Remote-Profiling-Elixir-Over-SSH/
https://drive.google.com/file/d/0Bz8Lmg2kodQiRXYwWVpGNXQtdG5hNG5GaDFFNF9UNXp4UXo4/view
http://learnyousomeerlang.com/distribunomicon
https://mfeckie.github.io/Remote-Profiling-Elixir-Over-SSH/
http://blog.lucidsimple.com/2016/01/10/connecting-two-nodes-in-elixir.html
https://bbhoss.io/remote-iex-access-via-ssh/
https://martinschurig.com/posts/2016/06/monitor-remote-elixir-phoenix-app-mix-task/
https://broot.ca/erlang-remsh-is-dangerous
http://blog.listincomprehension.com/2010/03/spoofing-erlang-distribution-protocol.html

Image from https://pixabay.com/en/cyber-security-protection-technology-3411476/