Resource quotas in GKE managed clusters are hard limits for the number of resources that can be created in a namespace. For a lot of cases you may not run into issues but the quotas are set low enough by default they may be run into for a cluster with more workloads. The quotas increase as you add more nodes to the cluster but for clusters with aggressive scale to zero or lots of small pods this increase may not happen. In this, we will go through how to increase or even remove the quotas from the cluster.

How to see the resource quotas

There is a resourcequota resource created in each namespace when you create the cluster. The default limits are:

  • ingresses.extensions: 100
  • ingresses.networking.k8s.io: 100
  • jobs.batch: 5k
  • pods: 1500
  • services: 500 You can use kubectl to see the resourcequota for a particular namespace:
$ kubectl get resourcequotas
NAME                  AGE   REQUEST                                                                                                                              LIMIT
gke-resource-quotas   14d   count/ingresses.extensions: 0/100, count/ingresses.networking.k8s.io: 0/100, count/jobs.batch: 0/5k, pods: 3/1500, services: 1/500

How to know if the quotas are causing problems

Within the cluster you will see an “exceeded quota” event similar to the following while attempting to create a new resource where the limit is full. This is when we try to create an extra my-service service when there are already 500 services on the cluster.

Warning  CreationFailed   2m35s (x13 over 2m59s)  route-controller  Failed to create placeholder service "my-service": services "my-service" is forbidden: exceeded quota: gke-resource-quotas, requested: services=1, used: services=500, limited: services=500

How to increase the resource quotas

To increase the quotas, you need to scale up the cluster to have more than 10 nodes. Once this is done the resourcequota will be increased, for example there will be a limit of 15k pods and 5k services. You can scale the cluster back down again if you do not need the extra nodes without and the increased resourcequotas will remain.

$ kubectl get resourcequotas
NAME                  AGE   REQUEST                                                                                                                               LIMIT
gke-resource-quotas   19h   count/ingresses.extensions: 0/100k, count/ingresses.networking.k8s.io: 0/100k, count/jobs.batch: 0/50k, pods: 1/15k, services: 3/5k

How to remove the resource quotas

According to the official GKE documentation, to remove the resource quotas you can scale the cluster to at least 100 nodes. In this case the quotas are removed entirely. Like above, they are not added back in once the cluster scales back down again.

An alternative to this is to contact GCP support to ask if the quotas can be removed without having to scale the cluster up.

Final Words

Here we went through how to increase and remove resource quotas in GKE clusters.