Most software developers agree: Distributed Computing and Clustering is hard. On the other hand, Erlang makes it extremely easy. But what can you do if you’re outside of the BEAM world?
You jump in a new hype instead of addressing the deeper problems! Kubernetes seems to be the new, shiny and agreed upon modern solution in town when you are talking about DevOps and Microservices in enterprises.
This article focuses on:
✓ My Kubernetes Experience
✓ Why Kubernetes is a better fit for large corporations with 100+ people.
✓ Experiment with something before adopting it
Table of Contents
Is Kubernetes a right fit?
Why Kubernetes Might not be for You
\t\t\t\t\t\t
\t\t TIP \t\t | Although it seems like a solution to solve all your needs it’s not. Don’t be tempted to think you need it, when in fact you probably don’t. The complexity and overhead it brings may only be worth in very complex applications in large (100+) teams. For a small team consisting of 1-3 people it’s simply not be worth it. |
My Experience
I’ve already seen the power of Elixir and Erlang when it comes to concurrency and distributed applications. Fault tolerance is the hallmark of the BEAM ecosystem. Yet I kept hearing all the people playing around with Kubernetes. At first when I compared what the benefits where, I couldn’t understand why people would go to such lengths to achieve something which is for free in the Erlang world.
As I often do, I could only form an opinion AFTER I’ve experimented with it. So I set on to do just that.
My Experience with Kubernetes was like falling in love
At the beginning, Kubernetes seemed to be the perfect fit for microservices and I was getting really excited about all the features it offers. You can really deploy applications much faster and deployments can be super fast.
✓ Rollbacks? Easy.
✓ Distribution? Check.
✓ Easy configuration? Check
✓ Steep learning curve, sure!
I’ve began experimenting with Kubernetes back in November 2019. Between December 2019 and February 2020 I ran it on a Linode VPS as a test.
I had been running k3s on a few Raspberry Pi between February and May 2020. I feel that I’ve explored it enough. My conclusions were definitive and we’ll get to them later.
What I did love about the Kubernetes ecosystem was the feature to simply deploy an app from a YAML template and have it get assigned a SSl certificate automatically, routing, load balancing. All done automatically! It seems like magic.
Not only that but It seemed to make the deployment process faster than other systems.
The Kubernetes Broken Heart
However, don’t let that "magic" fool you. It’s a steep learning and implementation curve. I’ve dedicated a huge amount of time learning about all of the small details which entail Kubernetes.
I could have instead worked on a variety of other projects. However, now I Know I won’t be missing anything.
There are many issues I’ve encountered and new problems which Kubernetes creates such as:
It treats each pod as cattle.. killing it whenever it doesn’t do it’s job or something goes wrong. Sure, we can have statefull sets but that brings wit hit another issue
Storage!! Storage in Kubernetes is a nightmare. k3s has a local path provisioner, but if you are in a cluster you’re really stuck. The solution would be to use various 3s party providers which is not acceptable in my situation since I want to store things locally. Another solution would be to use longhorn however that would provide with itself another problem… So all in all we’re stuck in a
As of writing this.. Deploying a database inside Kubernetes is a bad idea unless you can be sure your storage provider won’t screw things up. Few people have been able to successffully run SQL servers in kubernetes.
\t
\t
Even if there are many providers for Postgresql things can still go wrong.
\t
\t
\t
Do you know what everyone recommends? To follow the natural, most often mentioned alternative in how you can run PostgreSQL (read any SQL server) successfully. It;s to actually run outside of Kubernetes. But this is stupid! Why the heck would you go with all of the overhead of learning kubernetes just to jump out of it for your databases..
\t
Setup Kubernetes yourself? Good luck, see you next year.
\t
\t
K3s tried to make it easier, and they succeeded.. but it still came with a cost of bugs.
\t
Sometimes you need to have a private docker registry. Hosting it inside kubernetes is easy, however it can lead to issues if your cluster crashes..
Something clearly is wrong. There have been even more small things which buggered me like the fact that a very small change can propagate itself to crash your whole Kubernetes cluster.
And I was not taking into account the complexity of Kubernetes while getting started and setting up everything properly…
Although Kubernetes (k3s and k8s) is great in many aspects I’m beginning to sense that it might not be the best solution for sole developers or startups due to all of the overhead which it includes.
I Thought Kubernetes was a good fit for small businesses
I had even written a few articles I intended to publish around Kubernetes and self hosting, experiments, best practices. Aimed at Small I went so far in believing that Kubernetes was a solve all for small companies allowing them to automate faster and better.
While I still have a large amount of local unedited articles, I won’t pursue the path of editing and publishing them. Because I no longer believe Kubernetes is a solution for small development teams. Actually, I believe the opposite. It should be used there where it hurts the most. Big corporations which struggle with innovation and have a dedicated person.
So far I’ve only published Solving cert manager letsencrypt issues and How to Setup a persistent DNS with a dynamic IP.
What will I do in the future?
I think I’m going to switch back to using virtual machines and automated barebones deployments. Or probably a combination of virtual machines, bare metal & containers.
It’s pretty easy to scale a VM in any cloud provider up to 192 GB RAM and 32 CORES. which should be enough for a variety of even the most complex enterprise applications.
Adding a new machine whenever necessary.
Yes, this is how the Internet has been running for 30 years. If you use Erlang or Elixir as your stack then you’re covered.
I’ve looked at various alternatives to k3s, although most of them are again way too complicated, Docker Swarm and Nomad seem to have caught my attention, even Consul seemed allright.
The downsides is that you’re again locked into a system of complexities.
Building my own system
Will I be building my own system? Yes. I’ve started planning something geared towards the Elixir ecosystem. A system to just handle the basics and replace the need for a lot of manual work. It’s most likely going to be something which takes features out of software like Webmin, Virtualmin, Ansible, focused in easy deployment on VPS’es for developers.
We’ll see what grows out of it!
Conclusions
Never let peer pressure (colleagues, friends, companies, your boss..) guide you into adopting the wrong technology just because it’s shiny and new
Dedicate time to experiment with new technologies before you decide to use them while learning. You’ll save a lot of time and problems.
Building POC’s and discarding them is much easier than building a whole system and then deciding to migrate.
Adopt Elixir (BEAM, Erlang) if you’re seeking to build distributed, highly available, fault tolerant systems.