Kubernetes is among the most widely used container orchestration tools today. Kubernetes itself is an application that provides a framework and tools to manage, deploy, and scale container workload. It essentially leverages servers either on-premise or with cloud providers. Kubernetes manages our application, but we need a way to manage and install Kubernetes itself.

This is where Kubernetes distributions come into play. Distributions offer ways to install and manage Kubernetes in. The six most popular are xKS (EKS, AKS, GKS), OpenShift, Tanzu, or Rancher. Some distributions also come with additional software integrations to help handle tasks like CI/CD or monitoring.

This begs the question: which is better?

To help answer that question, this article will provide criteria to select a Kubernetes distribution; 8 perspectives we should consider so we can make an informed decision about which tool best fits your use case.

Kubernetes Platform vs. Distribution

Not all the tools we are looking at today are equivalent. It is important to define what we mean by a Kubernetes distribution.

Kubernetes Distribution: Offers an integrated solution for deploying and managing Kubernetes clusters; It handles everything from installation to configuration of Kubernetes components. They are package versions of the open-source Kubernetes project in the same sense as the Linux distribution.

Kubernetes Platform: These extend the distribution concept by adding pre-configured integrations and value-added services. Platforms are not just Kubernetes clusters as they offer a higher-level abstraction layer on top of the core components.

Let’s put the tools we are looking at in context:

• AKS, EKS, and GKE are Kubernetes distributions managed by their respective cloud provider

• OpenShift is a Kubernetes platform that leverages a distribution called OKD.

• VMware Tanzu is a Kubernetes platform.

• Rancher is a tool to manage Kubernetes clusters based on a distribution called RKE.

8 Considerations for Choosing the Right Kubernetes

1. Organizational maturity for cloud-native application deployment

Adopting Cloud-Native Applications is already an important change for an organization not limited to Container adoption. Adopting cloud-native applications means creating independently deployable, loosely coupled services.

This transformation is often achieved by adopting the DevOps automation and platform design culture that brings developers and operations together. However, it requires gaining sufficient expertise with new tools and Cloud platforms. The list of DevOps tools is now very long and hard to navigate.

As a result, when we get started, we may look for an opinionated solution where a minimal set of tools is provided to you and not just Kubernetes. OpenShift and Tanzu have an opinionated/prescribed way to help streamline the process. The two platforms provide an easier journey to the Cloud.

2. Availability of resources

Following the previous point choosing a platform often revolves around the workforce and skills available in our operation teams.

If we recently moved to the cloud or have been using the same Cloud provider for a while, we might consider using the Kubernetes distribution provided by your Cloud provider. In that case, one of the managed distributions, such as AKS, EKS, or GKE, might be tempting.

However, booting up a Kubernetes cluster from our Cloud provider is quite a heavy lifting. Once the Cluster is up and running, we still need to set up all the essential tools for monitoring, security, networking, etc. Additionally, we may want advanced features like a service mesh layer, event-driven auto-scaling, etc.

These criteria should make us lean toward either managed Distribution (AKS, EKS, GKE) or Platform (OpenShift, Tanzu). Rancher is more of an option for Kubernetes power users that want even more control. To ensure your cluster is secure, you might consider solutions like secret management service from Stakater.

3. Local development support

When choosing our platform, it is also important to consider the developer experience. An easily accessible local edition of our Kubernetes distribution or platform would enable a better understanding of Kubernetes by developers and local testing of important features.

OpenShift provides Minishift and OpenShift Local that emulate the platform for local development. Minishift targets platform engineers that need to implement or integrate with OpenShift. At the same time, OpenShift Local is designed for application developers looking to work as close as possible to their production environment.

Rancher offers Rancher Desktop to enable developers to work with a local Kubernetes cluster. This solution has become a famous replacement for Docker Desktop even by non-Rancher users.

When using the other distribution AKS, EKS, or GKE we still have the option of running a Kubernetes cluster using tools like Minikube or Kind. Also, when using a local Kubernetes cluster, we might want to look at tools like Tilt or Scaffold to provide the best developer experience. For more detailed guidance on setting up a test environment, you can refer to our blog on why you need a Kubernetes test environment and how to set it up.

4. Ease of deployment and setup

AKS, EKS, and GKE are made to be spawned easily. We could get a cluster up and running using our favorite Infrastructure as Code tools. Most of the Cloud provides Terraform modules to make management for the cluster simpler. With this solution, the effort is not on getting started but on getting the clusters configured with required tooling, setting up CICD for our workloads, hardening the cluster and generally getting it ready for production.

Rancher provides an easy deployment solution for deploying Kubernetes Cluster. Once the Rancher Manager is set up, getting the new Kubernetes Cluster up and running is a piece of cake. Compared to the xKS, we have to host and manage the Manager, but it is a small price to pay to manage many clusters across multiple cloud providers.

Getting OpenShift / Tanzu up and running may be a bit more involved. Of course, they are more complex as they contain more components. That being said, OpenShift also comes in a Managed version where RedHat will deploy OpenShift on the Cloud of our choice and manage it for us.

5. Ease of use

OpenShift, Rancher, and Tanzu provide a login-based console to manage clusters visually. This is a very nice addition when getting started that engineers appreciate greatly. With all three tools, we get a better user and operator experience via their dedicated UI. In contrast, AKS, EKS, and GKE offer limited dashboards and rely on their own policy and permission system to manage users.

OpenShift and Tanzu are solutions with a long list of out-of-the-box features, as already mentioned. There is a learning curve to getting started due to the layer of abstractions the platform provides. But once the developer is familiar with these concepts, they can quickly get started using the existing documentation provided by each tool.

On the flip side, xKS or Rancher can be adapted to any scenario, which gives you more flexibility but at the cost of doing more manual work and using third-party applications. This means that our platform team will have to document and educate the development team on how to use the new tools.

6. Continuous integration/continuous delivery (CI/CD) support

We have over a dozen CI/CD solutions available today. Choosing which tool to use and implementing a pipeline can be time-consuming. Another thing to look at is the support for GitOps. GitOps is a configuration paradigm where the state of our Kubernetes cluster is defined in a Git repository. Change to a cluster of resources and applications is dedicated by commits and merges on the repository.

OpenShift offers a ready-to-use solution OpenShift Pipelines, built on top of Tekton, an open-source Continuous Integration and Continuous Delivery pipeline engine leveraging Kubernetes style manifest to define our pipelines. Additionally, OpenShift comes out of the box with ArgoCD, a popular open-source GitOps tool.

Rancher provides Rancher Pipeline, which is essentially a wrapper around Jenkins. This provides a solution CI/CD for Rancher Users where the Jenkins instance is managed for us by the Rancher Manager. Rancher also provides a GitOps solution called Fleet used to keep our cluster in sync with our manifest in Git.

We need to choose our CI/CD tools chain with the other platform. As a side note, VMware Tanzu is maintaining Concourse CI, a somewhat unique solution.

7. Vendor lock-in consideration

Some customers prefer to use a vendor's supported distribution. This makes sense when you rely heavily on one vendor such as AWS, Azure, or GCP. Other organizations prefer the flexibility and power of open-source platforms that can be deployed in any cloud. But that also means that our workloads are tied into one provider and can be an issue if we intend to diversify at a later time.

With more vendors entering the container market, many will want to maintain their choice of a particular Kubernetes platform. Rancher offers the minimum constraints as it can deploy Kubernetes clusters in any cloud or on-premises. OpenShift and Tanzu also let us deploy to any Cloud provider. However, they both require us to use the ecosystem. This means that with OpenShift, we're relying on Red Hat, and with Tanzu, we are deploying VSphere. If we have a preference for an ecosystem in which to deploy, OpenShift or Tanzu are options worth exploring.

The most important thing is to ask if this vendor lock-in will be a problem in the future to minimize migration toil.

8. Consideration for on-premise or edge workload

If edge workload is important, we need a distribution that allows it. Typically xKS are primarily targeting cloud workload. However, major cloud providers understood customers' need to manage on-premise workloads. The catch is that we need to be careful about the requirement; AWS recently released EKS Anywhere, which allows bare metal servers or on-premise VM using VMware vSphere to be integrated into our cluster. GKE also leverages VMware vSphere for on-premise workload, while Azure relies on its own solution Azure Stack HCI and Windows Server to provide on-premise support for Kubernetes Clusters.

Needless to say, Tanzu supports on-premise workloads based on VMware vSphere. Additionally, Tanzu supports edge workload with VMware Edge Compute Stack. OpenShift also supports both on-prem and edge workloads. OpenShift achieves on-premise support thanks to the REHL ecosystem and the Hypervisor of our choice for our server farm. The OpenShift edge capability is provided by an Operator that enables us to manage edge resources the same way any Kubernetes resources would.

Rancher is, by design, a solution to deploy Kubernetes anywhere in the Cloud, on-prem, and to the edge. The particularity of Rancher is their lightweight Kubernetes distribution k3s that can be installed directly on an edge device.

Most distributions offer on-prem support, but we need to be careful about the requirement as all solutions have opinionated solutions to manage the op-prem workload. The same goes for edge support; while Tanzu, Openshift, and Rancher all support edge workload, they have different approaches to managing edge devices.

Conclusion

As a technology, Kubernetes is the platform of choice for orchestrating and managing container workloads at scale in today’s data-driven world. However, when it comes to selecting a distribution or platform, many considerations come into play which we discuss in this article.

The first distinction we need to consider is Distribution vs. Platform. This will greatly impact what we get out of the platform. The sad truth is that we can't just run our applications in production with Kubernetes alone. Choosing the right platform is always a trade-off between our technical requirements, our team's technical skill set, and the resulting cost.

If we are not up to the challenge of running a Kubernetes distribution or platform and don't want to reinvent yet another solution, there is another option available. Stakater App Agility Platform (SAAP) is an all-in-one OpenShift Managed Platform where you can focus on building your app and delegate the hassle of managing the platform to Stakater.