This is an ATTENTION to all Elixir/Phoenix/Erlang users out there that
use distributed nodes or software that uses nodes. Whenever I talk about
Erlang or Elixir you can interpret both since it’s the same underlying
system at work.
I’m not talking about node.js. But about the distributed OTP programming
system that beats the hell out every other programming language.
All programming languages want to do distributed programming like Elixir
and Erlang do when they grow up. I’ve recently started implementing
distributed OTP systems and I’m still awed by the powers of the Elixir,
Erlang and OTP ecosystem!
NOTE: Windows users… you might have to figure out how to do it with
Putty:) There is an inherent security "flaw" which could lead to big
problems if let unattended. Read below for more info
In November 2017 I discovered Elixir which is based upon Erlang. I fell
in love, and from the first moment, I read the
Elixir
Getting Started guide (.EPUB file download) I felt something
mysterious.
Reading the code even though it was for the first time AND it’s a
functional language made it feel right.
I’ve never felt this before with any other language except Tcl which I
grew up with.
You
can
read
more
about it in
its
own
special article.
Erlang and Elixir is a great ecosystem. I’ve been told that before and I
didn’t believe it until i started reading code and testing things out
myself.
Before I wanted to start out creating production software I wanted to
test out its security potential.
Compared to other programming languages and tools it is very robust.
You probably won’t find overflow problems, stack smashing is impossible
since everything is held in the HEAP and the stack is simulated.
Command injection is possible but that depends on the app itself.
One other issue I’ve detected is the usage of :erlang.term_to_binary
and :erlang.binary_to_term
. This is a way to serialize data to
binary when sending it over the wire or saving it in files.
It’s an alternative to XML, JSON etc. It can be used for blockchain
encryption and stuff.
Since internally this uses zlib compression and it’s used quite a lot..
it can bring problems like binary zip bombs.
Why? Well, everything is stored in memory. And when decompressing a
binary bomb that is only 4 MB you can use up to 4 GB of memory.
But that’s not the problem of the day.
Erlang Node Seurity
Erlang and Elixir’s distributed node system has another inherent weakness. It’s cookie based.
It’s advised that you don’t run Erlang on internet facing websites.. But with the rise of Elixir and Phoenix this becomes quite complicated.
One of the first problems is that by default the Distributed Node system
works in plaintext.
Another problem is that node authentication is cookie based which is
deterministically generated.
All nodes share the same cookie. So if a node is compromised, all your
systems can compromised. Now, meaning that the attacker knows these
things and is an Erlang fan. But we don’t want to take any chances, do
we?
+
Compromised how?
Well, you can run code on the operating system via the :os.cmd() function. You can open a remote shell from within Erlang/Elixir itself. Another way is that you can actually compile code across the distribution via the 1 node.
One security researcher pointed out that you can actually guess a cookie because of the way atoms work in Erlang.
You can view a full talk about the issue here:
https://www.youtube.com/watch?v=42k70Y-yTYY
So basically atoms are limited in Erlang (and elixir) he claims to have
tested this and that he can guess the cookie atom based on a connection
he can make using a node name
This is interesting in 2 ways:
You could guess the cookie by depleting the atom resources
You can CRASH a VM if it tries to recreate all the atoms
Both these things are potentially a problem
Another problem is the usage of the elixir iex --remsh or remote shell
option.
Because the way of how the distributed node system works as explained
earlier you can have problems on your OWN pc when debugging.
If you connect with your development computer to a compromised node
via --remsh you become part of the node, so in theory and practice
the node you’ve connected to CAN run code on YOUR PC.
This means that your private files and even SSH keys can be read.
Ouch, this hurts!
Potential Solutions
Don’t use Distributed Nodes
Well, if you only have 1 system.. You could just.. not use the
distributed node system and you’re safe.
This means that you should never use the Elixir --sname or --name
function unless you are willing to risk things..
But Elixir and Erlang is BUILT for distribution and everything is
internet connected today. This means that any startup probably runs
their VM’s in the cloud and needs to interconnect them.
Ok, so the next logical thing would be to implement something similar in
Elixir and use it.
This is not such an easy task since it would mean to reimplement the
wheel.. And it would probably NOT be as useful as the node issue.
If you don’t need the power of distributed nodes then don’t use them.
If you are willing to write your own distribution mechanism then you can
implement security in any way you want.
As mentioned earlier, maybe it’s not a good idea to reimplement the
wheel.
However… even the simple usage of :observer.start REQUIRES the usage
of the cookie-based node system.
So when you try to "debug" your Phoenix app.. Heh.. Guess what, you’ve
opened up the door to the unknown.
But let’s be serious, why use Elixir or Erlang if you are not thinking
about using it’s full potential and aren’t into distributed computing?
Setting up Erlang to use TLS/SSL
A solution to the problem can be to setup Erlang to be TLS/SSL based.
Yaay!
This can be done, it will be a tedious setup since you have to setup
certificates with public key encryption..
Be sure that all your nodes are up to date.
But you then still need to secure epmd which will still continue to
talk UNENCRYPTED.
Securing your certificates and server is ALSO important. This, however, CAN and SHOULD be automated.
Using SSH
This solution will solve the problems explained earlier except for the `+term_to_binary +`issues which are up to the developers to secure.
My recommendation is to setup a firewall that blocks everything. Setup SELINUX if required.
Then, when starting up any Erlang or Elixir based apps
Define the full name --name yourname@127.0.0.1
BE SURE to put the machine name of the node as 127.0.0.1 (0r 127.0.2.4
or whatever in the 127.0.0.1/8 range) FOR ALL your NODES.
This will make sure that when someone tries to connect to it.. they will
be able only if they’re local hosted. I know you might find this
slightly backward but read on.
Whenever a node needs to connect to another node setup a SSH reverse
port connection to an low privileged user.
ssh -L4369:localhost:4369 -L39974:localhost:39974
edge@node2.host.example
This sets up a local port on your machine on localhost which then connects to the server and tunnels everything directly. The 4369 port is for the EPDM service on the remote server. You will probably have to setup multiple and different local ports for it since each node has it’s own epdm service on 4369. SSH easily supports this.
Then connect via the local port. This will have 2 benefits.
First, you’ll have encrypted traffic. Second your ports will ONLY be
available on the nodes themselves. Two birds one stone. The only
downside is that you’ll have to write some scripts to automate the SSH
port forwarding. Also take care to implement authentication via public
key systems and don’t pass passwords around. This will involve a little
bit of overhead but from a security standpoint it’s worth it.
Note | if you want to use the observer functionality you will need Xforwarding |
# ~/.ssh/configHost ci Hostname 192.168.1.100 User username ForwardX11 yes ForwardX11Trusted yes
Then whenever you connect to the node, you do it via the local FQDN.
Even for debugginv via --remsh you first open a ssh -L connection to
localhost AND connect to the localhost port.
The setup of this process can be automated.
Oh and one more thing. If you ARE using Nodes with cookies
Read more on the erlang:set_cookie/2 and/or Node.set_cookie . You can
set different cookies for each node.
This will minimize the problem if one node is compromised since the
attacker won’t be able to connect to the other ones
At least untill he figures a way to get the cookies of the other nodes.
To sum it up
Don’t use --sname --remsh --name on public facing computers UNLESS
you’ve setup your SSH, firewall and other security systems.
Have different cookies
Use ssh for security
Resources to help you understand more
You can view a full talk about the issue here:
https://www.youtube.com/watch?v=42k70Y-yTYY
https://mfeckie.github.io/Remote-Profiling-Elixir-Over-SSH/
https://drive.google.com/file/d/0Bz8Lmg2kodQiRXYwWVpGNXQtdG5hNG5GaDFFNF9UNXp4UXo4/view
http://learnyousomeerlang.com/distribunomicon
https://mfeckie.github.io/Remote-Profiling-Elixir-Over-SSH/
http://blog.lucidsimple.com/2016/01/10/connecting-two-nodes-in-elixir.html
https://bbhoss.io/remote-iex-access-via-ssh/
https://martinschurig.com/posts/2016/06/monitor-remote-elixir-phoenix-app-mix-task/
https://broot.ca/erlang-remsh-is-dangerous
http://blog.listincomprehension.com/2010/03/spoofing-erlang-distribution-protocol.html