Kubernetes Hacks and Tricks – #8 Pod Preemption, Eviction and Priority classes

In the previous article, we learnt about QoS classes in Kubernetes and how Kubernetes evicts Pods if node resources are running out. Now it’s time to be familiar with the PriorityClass resource in Kubernetes. Imagine you have a bunch of Pods with the same QoS class in the worker node, and the worker node’s resources are running out! How should Kubernetes decide which Pod should be evicted to free up node resources? Here is where that PriorityClass comes into the picture.

Follow our social media:

https://www.linkedin.com/in/ssbostan

https://www.linkedin.com/company/kubedemy

https://www.youtube.com/@kubedemy

https://telegram.me/kubedemy

Priority class lets you prioritise your workloads and tell Kubernetes which ones are more important and which ones should be evicted first in the time of node pressure.

Read the previous article about Kubernetes QoS classes:

Kubernetes Hacks and Tricks – #7 Pod and Container Resource and QoS classes

In Kubernetes, PriorityClass has two use cases:

1- Helps kube-scheduler to evict low-priority Pods when a high-priority Pod is waiting in Pending mode in the queue due to lack of resources.

2- Helps kubelet to evict Pods when the node is under pressure and other high-priority Pods need more resources to do their job. Note that there is a hidden rule in this type of eviction that will be discussed later in this article.

Default Kubernetes Priority classes:

Kubernetes has two default system priority classes to save cluster and node critical workloads in pressure situations.

Cluster-critical priority class should be used when the location of the application does not matter. For example, the Prometheus monitoring system.

Node-critical priority class should be used for all applications that need to be run on specific nodes only. For example, Node-exporter, CNI node plugin, etc.

Create your own PriorityClass:

PriorityClass name cannot be started with system- as it is reserved for Kubernetes itself. Value can be -2147483648 to 1000000000. Higher value, higher priority.

The default priority class is used for all the Pods without priorityClassName option.

When preemptionPolicy is set to Never, the scheduler will not evict any other Pods to schedule pending Pods of this priority class.

How to use PriorityClass:

If you don’t assign any priorityClassName to the Pod, the global default priority class will be used for the Pod automatically.

Important note about node pressure eviction:

This is the hidden rule!

kubelet node-pressure eviction does not evict Pods when their usage does not exceed their requests. If a Pod with lower priority does not exceed its requests, it won’t be evicted, and a Pod with higher priority that exceeds its requests may be evicted.

Conclusion and Best Practices:

Understanding Kubernetes scheduling and the kubelet eviction process can help you deploy reliable and highly available workloads on Kubernetes. For every workload, you need to use proper QoS and Priority classes. For all cluster and node-critical workloads, you must specify system priority classes created by Kubernetes. Kubernetes has not defined any default priority class, and you need to create your default priority class.

If you like this series of articles, please share them and write your thoughts as comments here. Your feedback encourages me to complete this massively planned program.

Follow my LinkedIn https://www.linkedin.com/in/ssbostan

Follow Kubedemy LinkedIn https://www.linkedin.com/company/kubedemy

Follow Kubedemy Telegram https://telegram.me/kubedemy