AWS EKS – Part 16 – Enable Secrets Encryption at Rest with AWS KMS Service

Encryption At Rest, EAR, is a term referred to as encrypting data when they are stored on disk. In Kubernetes, all objects and resource manifests are stored in the etcd database, and they don’t get encrypted by default, so if someone gains access to the database, all data can be read in plaintext format. It’s not a problem to see resources like Pod, Deployment, Service, ConfigMap, etc., in plaintext format from the database as we don’t put sensitive data in them, but Secrets are entirely different. Keeping them safe is super important as they consist of application credentials, node bootstrap tokens, service account tokens, external service credentials, etc., and if we store them in plaintext and someone gains access to the database, we will lose everything.

Important: Encryption at Rest is not enabled by default in EKS clusters.

Follow our social media:

https://www.linkedin.com/in/ssbostan

https://www.linkedin.com/company/kubedemy

https://www.youtube.com/@kubedemy

https://telegram.me/kubedemy

Register for the FREE EKS Tutorial:

If you want to access the course materials, register from the following link:

Register for the FREE AWS EKS Black Belt Course

Deep dive into the ETCD database:

In a cluster without Secret encryption, secret resources are saved in plaintext in the database. To get objects from etcd database, you can run the following command:

ETCDCTL_API=3 etcdctl \
  --cacert ETCD-CA.crt --cert ETCD-CLIENT.crt --key ETCD-CLIENT.key \
  --endpoints https://ETCD-SERVER:2379 \
  get "/registry/secrets/NAMESPACE/NAME"

As you can see, we got the manifest and decoded the secret.

In a cluster with Secrets encryption enabled, if you try the same command, you don’t get the same result, and you will get the encrypted version of the object, which can only be decrypted by kube-apiserver on-fly and in the runtime mode.

EKS and ETCD Database Access:

In AWS EKS, etcd is provided as a managed service, and no one except AWS can access it. Although no one can access it, if someone gains access to AWS infrastructures, your secrets can be exposed to the wrong persons. So, it’s better to enable encryption at rest, at least in production environments. To enable Encryption at Rest in EKS, AWS provides a way to integrate EKS with KMS so that the kube-apiserver can read KMS keys to encrypt Secrets before writing them to etcd and decrypt them after reading them from the database, which keeps secrets safe.

Enabling Encryption at Rest setup procedure:

• Create a new KMS customer-managed key for clusters.
• Enable Encryption at Rest for fresh cluster during deployment.
• Enable Encryption at Rest on existing EKS clusters.
• Encrypting existing Secrets in the EKS cluster.

Step 1 – Create KMS Key for EKS Cluster:

The KMS key for encrypting Kubernetes Secrets should be symmetric and in the same region as the cluster. To create a new key, run this command:

aws kms create-key \
  --tags TagKey=owner,TagValue=kubedemy \
  --description "Kubedemy cluster Secrets encryption key" \
  --no-multi-region \
  --key-usage ENCRYPT_DECRYPT \
  --key-spec SYMMETRIC_DEFAULT \
  --origin AWS_KMS

Step 2 – Create EKS Cluster with Encryption at Rest:

To learn more about how to deploy an EKS cluster, read the following articles. The difference between the following command and the typical cluster creation command is that we enable Encryption at Rest for the cluster to encrypt secret resources.

AWS EKS – Part 2 – Deploy managed cluster control plane

AWS EKS – Part 12 – Deploy fully air-gapped EKS clusters without internet

aws eks create-cluster \
  --name kubedemy \
  --role-arn arn:aws:iam::231144931069:role/Kubedemy_EKS_Cluster_Role \
  --resources-vpc-config subnetIds=subnet-0ff015478090c2174,subnet-01b107cea804fdff1,subnet-09b7d720aca170608,endpointPublicAccess=true,publicAccessCidrs=0.0.0.0/0 \
  --kubernetes-network-config serviceIpv4Cidr=172.20.0.0/16,ipFamily=ipv4 \
  --encryption-config resources=secrets,provider={keyArn=arn:aws:kms:eu-west-2:231144931069:key/8f917ca2-7003-41fc-b2a0-b95b3e0f3027} \
  --kubernetes-version 1.28 \
  --tags owner=kubedemy

As the cluster is fresh, all secrets added will be encrypted automatically.

Step 3 – Enable Encryption at Rest on existing clusters:

To enable Encryption at Rest on an existing cluster, you must create a KMS key and update the cluster configuration. Once the encryption configuration is in place, new secrets will be encrypted by the KMS key before being stored in the etcd database.

aws eks associate-encryption-config \
  --cluster-name kubedemy \
  --encryption-config resources=secrets,provider={keyArn=arn:aws:kms:eu-west-2:231144931069:key/8f917ca2-7003-41fc-b2a0-b95b3e0f3027}

Step 4 – Encrypting existing Secrets:

After enabling Encryption at Rest on an existing cluster, new secrets will be encrypted automatically, but all existing secrets must be updated to become encrypted.

To trigger encryption on existing secrets, run this command:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Important Considerations:

• Once you enable Encryption at Rest, you can’t disable it.
• If you delete the KMS key, you will lose the cluster and can’t recover it.
• To minimize KMS key access, you can add resource-based policies.

Conclusion:

Although AWS services are completely secure by design, you should always keep your data safe. AWS provides a shared responsibility model for security, which means you are responsible for security in the cloud and the security of your resources.

If you like this series of articles, please share them and write your thoughts as comments here. Your feedback encourages me to complete this massively planned program. Just share them and provide feedback. I’ll make you an AWS EKS black belt.

Follow my LinkedIn https://www.linkedin.com/in/ssbostan

Follow Kubedemy LinkedIn https://www.linkedin.com/company/kubedemy

Follow Kubedemy Telegram https://telegram.me/kubedemy