Kubernetes itself is great for scheduling within a single cluster but if we have many clusters, spread across geographies, we start to need something above the cluster level. This is where Kubernetes Multi-Cluster comes in in allowing us to manage apps across clusters.
Ask nearly anyone who uses kubernetes what they prefer between treating applications like cattle or pets nearly all will say cattle. Pets require special individual attention but cattle are managed within herds, at least that’s what the metaphor explains. In kubernetes land it means using a deployment or daemonset to create your pods and let kubernetes take care of the rest with things like scheduling and restarting.
This is easy to manage for a single cluster but becomes increasingly difficult as the number of clusters scales up. Creating the same deployment on each cluster becomes tedious and error prone.
In this post I’m going to give the situation where an internal platform team creates and manages clusters with some base level of services in it, for example logging, metrics and ingress along with a few others. We then allow other teams to deploy their apps to this cluster.
For our own services and a low number of clusters the most straightforward way would be to hook up some CICD system like jenkins to be able to deploy to each cluster. There may be some unique config per cluster so each app would get templated out and then we could use something like helm to generate the config as we deploy. However this process becomes unwieldy as the number of clusters increases as for each one it has to be connected to jenkins, a new step added to the pipeline and then the unique config added.
Even for other teams apps it’s not straightforward to deploy. We could allow them just hook up their own jenkins pipelines but what if we want to do the deployments too? We can’t expect each team be experts in kubernetes so the more we do to help them the better. Just using jenkins would require each one to be hooked up individually requiring a lot of time for each app and cluster.
All this sounds like we’re treating the clusters more like pets than cattle. Each service and cluster is bespoke and needs unique attention. Ideally we’d let something else handle all this repetitive work and allow us to focus on adding features. This is where kubernetes multi-cluster comes in to help. Different types help for different things as I’ll lay out below as each was designed for different use cases. What we’re looking for is to deploy our common config to every cluster and then deploy other teams apps depending on what they have configured.
GitOps
For the common config we’d like to put on each cluster then one of the most suited options is to use some gitops tool. How these tools operate is they use git as a source of truth for what should be deployed. If we specify what should be installed, it templates out the yaml for each of the apps and deploys it. This is useful if we’ve lots of clusters, say 100, and we need the same logging and monitoring applications on each cluster. What gitops tools do not do well is dynamically deploy applications based on factors such as load but we’ll get to more suited tools later. One of the most well known gitops tools for kubernetes is Argo.
For the other team’s apps we’ve got a number of things we could do. We could use gitops again to pick up all the other apps but like I said earlier if we have to dynamically figure out where an app should go then gitops is probably not the way to go.
Kubefed
Instead we could use a tool based on kubernetes federation, kubefed for short. The first version of kubefed (v1) was designed for services but a second version (v2) allowed for multiple kinds of objects and allowed the control plane of one cluster to manage other clusters. All you’d have to do is deploy apps to the control cluster and kubefed would take care of putting them on the managed clusters.
Originally this was a kubernetes official project but the official repo was archived January 2023. Instead there are a number of offshoot projects that take inspiration from it. Karmada is the closest to what kubefed was trying to do but aims to be a more plug and play plugin. Open Cluster Management on the other hand is more of an API that allows third parties to implement for them to orchestrate across clusters in turn.
Networking
One thing about the ones above is they are for deploying apps to the clusters. Once the apps are there they continue to behave as though they are on completely separate clusters with nothing to do with each other. However there are some cases where you want the apps on the clusters to talk to each other, or otherwise control the communications between them. For this you would need a network connection. There are options like CiliumMesh or Submariner with the option you choose based on your specific circumstances.
Virtual Kubelet
Finally we have virtual kubelet projects which while made for something else have been co-opted for multi cluster management. What this is is a virtual kubelet on each of the clusters that is controlled by the central control cluster. Each cluster shows as a node on the control cluster. The virtual kubelet coordinates the pods amongst the clusters, deploying and scheduling depending on availability.
This has been a very brief introduction on why you may want to use multi cluster in kubernetes along with what the different types of projects that are available. In the future I will go into each in more detail.