👋 Hey there, this is quite an old article. Since I have found fly.io to be the one-stop-shop for all my PaaS related needs. Their free pricing tier is quite nice, the docs are thorough and their pricing scales quite nicely. Be sure to check them out!
It starts off small, just trying to nudge the team into the direction of a new tech or small module in a project and before you know it, you’re neck deep in fancy docs of some weird project that solves your business case in just 3 easy steps! Trying to find a way to do that trivial thing you want to however, seems to take a lot more steps.
We all have those moments; one of those moments for a lot of people in the last year or so has been a project named Kubernetes. Some companies and teams have found a great use for it, some haven’t and burned themselves in the process. I’m somewhere in the middle, just making a huge contraption to eventually burn myself with. Until then, let’s take a look on how to deploy a simple PaaS-like platform on top of k8s.
Well, where do we start? There must be a simple way of finding something like this, right? Maybe, so let’s start with a simple DuckDuckGo search.
So where do we go next? Well let’s do some searching on HackerNews! Okay, my first query reveals one dead article. That’s not good. Ah, just removing some words does it a lot better. I also stumbled on a nice awesome-list on GitHub that was a huge help.
After more extensive searching, I ended up making a list of possible projects as candidates for my use case:
There are a lot more alternatives, some of which I tried, some of which didn’t show a lot of activity on their repos, some of which were clearly meant for big enterprise companies etc.
So what’s this use case? At Quest Vault we currently run our e-commerce website on a basic DigitalOcean droplet with a simple Wordpress installation. Although keeping this installation running through some simple bash scripts for deployment and running a copy of the server locally for testing/staging is doable, I wanted to build a platform on industry tech instead of some bash scripts. Making these bash scripts is fun and having your own deploy stack is neat, but having a standard and not having to worry about the tools that you’re using is a luxury I wanted to have for Quest Vault.
For now, I wanted to test out these projects on a garbage server I have at my office that runs k3s. K3s has a reverse proxy to my DigitalOcean droplet instead of being accessible on the internet. This means that the project should support on-premise deployment.
Another thing that I wanted was a complete abstraction away from k8s. This means that I don’t want to deal with a lot of yaml’s or with deploying helm charts all the time, I want to think in terms of my application and have it be doable through a CLI. In short: I just want to push a button and have it work.
Our applications have a lot of moving parts, some of them are just simple scripts, some of them are larger applications providing communication to our game clients. Whatever they are, our platform needs to support a wide array of different application types. This usually means support for deploying through a **Dockerfile**.
Most of the applications we plan to run are closely bound to state. With Wordpress for instance, we need to have a place to store pictures. We also have a lot of photo assignments in-app that require storage. We need a way for our applications to have some form of persistence.
I like a lot of projects, but what sets apart a good project from a great project is community and industry adoption. There’s close to no difference between having a your own bash script that no one knows about or a project on GitHub with 3 active users. If you’re screwed or need some advice for whatever reason, you want be able to count on an active community to help.
My experience with Knative started off great! Reading about it made me happy to learn that I was able to run a platform that Google uses for their own PaaS-like deployments inside of their platform. Given that Google made k8s, this must be a perfect fit! Trying to install it ended up being more difficult than expected. It seemed there was no easy way of installing this platform, and not being able to use a platform with ease is a risk in the future. Perhaps it’s just me, and maybe I should’ve read deeper into everything Knative, but this made me move on to the next alternative.
Super easy to install! I was quickly able to start running this platform. It ticked off most of my boxes but seemed more of a fun way of implementing OpenFaas rather then being a fully fledged PaaS alternative. I had difficulty seeing how to put our use case into this particular platform. If you’re using more loosely coupled projects or smaller functions together, this is a great choice! Maybe I should look at it another time, but for now, I was already looking for another alternative.
Convox looked great! A few ex-Heroku engineers, building a platform on top of k8s? It seemed perfect! I wanted to try it out and I quickly started deploying it on a DigitalOcean k8s cluster. The developer experience was great! However, they didn’t seem to support an on-premise version of the platform. Also, the project didn’t seem to have a very big community outside of some early adopters. The relative unknown-ness of this project ended up with me dropping it and looking for another alternative.
This is a really cool project. I love it when a small independent company develops an innovative solution. It was a breeze to setup and their methodology is a good abstraction from k8s, but they also let you keep some form of control through classic k8s ways, like yaml files. I was really happy implementing this, and it worked out great! I did, however, notice some rough edges around some of the CLI, but I think these were just smaller hickups and not representative of the final product.I unfortunately felt an itch to look for a final (and last) alternative.
Wow, this project ticks all the boxes. A really easy to use CLI? Check. No longer having to interact with k8s in any way? Check. Using Dockerfiles for deployment? Check! They also come with a large list of features that other platforms didn’t implement, or implemented poorly. Rio being from Rancher, it seems to have a great amount of support from their active community.
I quickly setup the reverse proxy to my k3s instance and started setting up Rio. Through the quickstart on their GitHub page it ended up being super easy:
# Setting up the reverse proxy to k3s
ssh -nNTL 6443:localhost:6443 droplet &
# Installing Rio
curl -sfL https://get.rio.io | sh -
# Running the example project
rio run https://github.com/rancher/rio-demo
And that’s it! I’m super excited to see if our existing infrastructure migrates with the same ease.
The default installation of Rio allows you to use their rDNS service on-rio.io which is really cool, but unneeded for a garbage server behind a reverse proxy. I’m also not experienced with using Linkerd yet, so I’m just disabling that for now. Re-installing Rio with rio install --disable-feature rdns,letsencrypt,linkerd gave me the result I wanted.
Next up, installing a custom ClusterDomain through kubectl allowed me to use another domain than on-rio.io. I ended up instaling dnsmasq and making a fake domain called app.rio where my apps would resolve on. This would allow me to easily test connectivity to the apps on the garbage server itself.
apiVersion: admin.rio.cattle.io/v1
kind: ClusterDomain
metadata:
name: app.rio
spec:
httpPort: 80
I still had to make a way for me to connect to this cluster from my DigitalOcean droplet. I did this by reverse proxing port 80 from garbage server to my droplet on port 8080. Port 80 was used by Rio installed Gloo’s gateway-proxy.
As a final step, I setup my nginx configuration to point to the Gloo gateway:
server {
listen 80;
server_name your.domain.name;
location / {
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_pass http://localhost:8080;
}
}
Two important things to note here are the proxy_http_version 1.1 and the proxy_set_header Host.
proxy_http_version is important because Envoy based Gloo doesn’t support gatewaying on http_version 1.0, but only on 1.1. Otherwise, it just returns a 426 Upgrade Required error. The Host header is important for the implementing a PublicDomain. To add a PublicDomain, it’s important that this matches the server_name or proxied Host header, otherwise Gloo can’t recognize which service I’m trying to reach.
rio domain register your.domain.name rio-demo
And that concludes my adventure on finding the most fitting PaaS solution built on Kubernetes.
Thank you so much for reading! 👋