In the previous article, we learnt about QoS classes in Kubernetes and how Kubernetes evicts Pods if node resources are running out. Now it’s time to be familiar with the PriorityClass resource in Kubernetes. Imagine you have a bunch of Pods with the same QoS class in the worker node, and the worker node’s resources are running out! How should Kubernetes decide which Pod should be evicted to free up node resources? Here is where that PriorityClass comes into the picture.
1- Helps kube-scheduler to evict low-priority Pods when a high-priority Pod is waiting in Pending mode in the queue due to lack of resources.
2- Helps kubelet to evict Pods when the node is under pressure and other high-priority Pods need more resources to do their job. Note that there is a hidden rule in this type of eviction that will be discussed later in this article.
Default Kubernetes Priority classes:
Kubernetes has two default system priority classes to save cluster and node critical workloads in pressure situations.
Cluster-critical priority class should be used when the location of the application does not matter. For example, the Prometheus monitoring system.
Node-critical priority class should be used for all applications that need to be run on specific nodes only. For example, Node-exporter, CNI node plugin, etc.
Create your own PriorityClass:
PriorityClass name cannot be started with system- as it is reserved for Kubernetes itself. Value can be -2147483648 to 1000000000. Higher value, higher priority.
The default priority class is used for all the Pods without priorityClassName option.
When preemptionPolicy is set to Never, the scheduler will not evict any other Pods to schedule pending Pods of this priority class.
How to use PriorityClass:
If you don’t assign any priorityClassName to the Pod, the global default priority class will be used for the Pod automatically.
Important note about node pressure eviction:
This is the hidden rule!
kubelet node-pressure eviction does not evict Pods when their usage does not exceed their requests. If a Pod with lower priority does not exceed its requests, it won’t be evicted, and a Pod with higher priority that exceeds its requests may be evicted.
Conclusion and Best Practices:
Understanding Kubernetes scheduling and the kubelet eviction process can help you deploy reliable and highly available workloads on Kubernetes. For every workload, you need to use proper QoS and Priority classes. For all cluster and node-critical workloads, you must specify system priority classes created by Kubernetes. Kubernetes has not defined any default priority class, and you need to create your default priority class.
If you like this series of articles, please share them and write your thoughts as comments here. Your feedback encourages me to complete this massively planned program.