Kubernetes: Cleaning Up Replica Sets After Deployment Updates

• Introduction
• Step-by-Step Guide
• Code Example
• Additional Notes
• Summary
• Conclusion
• References

In Kubernetes, Deployments provide a declarative way to manage your application's lifecycle. They ensure that a specified number of Pods, representing your application instances, are running at any given time. Deployments achieve this by using ReplicaSets, which are responsible for maintaining the desired number of Pods. When you update a Deployment, it doesn't directly modify the existing Pods. Instead, it creates a new ReplicaSet with the updated configuration and gradually replaces the old Pods with new ones managed by the new ReplicaSet.

Kubernetes Deployments use ReplicaSets to manage your Pods. When you update a Deployment, it creates a new ReplicaSet and scales it up while scaling down the old one.

To control how many old ReplicaSets are kept, use .spec.revisionHistoryLimit in your Deployment definition.

apiVersion: apps/v1
kind: Deployment
...
spec:
  revisionHistoryLimit: 3 

This example keeps a maximum of 3 old ReplicaSets. Setting it to 0 will remove all old ReplicaSets immediately.

However, simply deleting ReplicaSets directly is not recommended. Kubernetes manages these for rollback functionality and history. If you're experiencing issues with old ReplicaSets not being cleaned up, ensure your Deployment configuration is correct and that there are no other conflicting factors in your cluster.

This code provides an example of a Kubernetes Deployment configured to maintain a revision history limit of 3. The deployment.yaml file defines the deployment, while the update_deployment.sh script simulates updates to the deployment, showcasing how old ReplicaSets are managed based on the revision history limit. The script updates the deployment five times, each time updating the Nginx image tag and listing the associated ReplicaSets. The output demonstrates that Kubernetes retains a maximum of three old ReplicaSets, deleting older ones as new updates are applied.

This example demonstrates a Kubernetes Deployment with a revisionHistoryLimit set to 3. It also includes a script to simulate updates and demonstrate how old ReplicaSets are managed.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest

update_deployment.sh

#!/bin/bash

for i in {1..5}; do
  echo "Updating deployment to version $i..."
  kubectl set image deployment/nginx-deployment nginx=nginx:$i --record
  sleep 5
  kubectl get replicasets -l app=nginx
  echo "-------------------------"
done

Explanation:

1. deployment.yaml: This file defines a Deployment named nginx-deployment with revisionHistoryLimit set to 3. This means Kubernetes will keep a maximum of 3 old ReplicaSets after each update.
2. update_deployment.sh: This script simulates updating the Deployment 5 times.
   • It uses kubectl set image to update the Nginx image tag, effectively creating a new revision each time.
   • The --record flag annotates the Deployment with information about each update.
   • It then uses kubectl get replicasets to list the ReplicaSets associated with the Deployment, showcasing how old ReplicaSets are managed based on the revisionHistoryLimit.

Running the example:

1. Save the deployment.yaml file.
2. Save the update_deployment.sh script and make it executable with chmod +x update_deployment.sh.
3. Apply the Deployment: kubectl apply -f deployment.yaml
4. Run the update script: ./update_deployment.sh

Output:

The script will output the list of ReplicaSets after each update. You will observe that Kubernetes keeps a maximum of 3 old ReplicaSets, deleting older ones as new updates are applied.

Important Note:

While this example demonstrates how revisionHistoryLimit works, remember that directly deleting ReplicaSets is not recommended. Let Kubernetes manage them for rollback functionality and history. If you encounter issues with old ReplicaSets not being cleaned up, review your Deployment configuration and investigate potential conflicts within your cluster.

Understanding the Relationship:

• Deployments are like blueprints for your desired state. They tell Kubernetes what you want running, not how to run it.
• ReplicaSets are the workers that translate the Deployment's blueprint into actual Pods. They handle scaling and ensuring the desired number of Pods exist.
• Revision History is managed through old ReplicaSets. This allows for easy rollbacks to previous versions of your application.

Revision History Limit:

• Purpose: Prevents unlimited accumulation of old ReplicaSets, which can consume resources and clutter your cluster.
• Trade-off: A lower limit saves resources but reduces rollback options. A higher limit provides more rollback flexibility but uses more storage.
• Best Practices:
  • Start with a reasonable default (e.g., 3-5) and adjust based on your application's update frequency and rollback needs.
  • Consider external tools for long-term historical data if you need to retain information beyond the limit.

When to Investigate Potential Issues:

• **Old ReplicaSets persisting despite revisionHistoryLimit: **
  • Verify the Deployment configuration is applied correctly (check for typos, syntax errors).
  • Investigate potential conflicts with other controllers or custom resource definitions (CRDs) that might be managing the ReplicaSets.
  • Check for errors or warnings in the Deployment and ReplicaSet events.
• Resource constraints: Ensure your cluster has enough resources to create new ReplicaSets and Pods during updates. Insufficient resources can lead to unexpected behavior.

Important Considerations:

• Directly deleting ReplicaSets is generally discouraged. Kubernetes uses them for rollbacks and history. Manual deletion can disrupt these functionalities.
• Resource cleanup: While Kubernetes manages ReplicaSets, consider periodically cleaning up unused Deployments and other resources to maintain a clean and efficient cluster.
• Monitoring: Monitor your Deployments and ReplicaSets to ensure they are functioning as expected and to identify any potential issues early on.

By understanding the interplay between Deployments, ReplicaSets, and revision history, you can effectively manage your application deployments and rollbacks in Kubernetes.

Kubernetes Deployments and their management of ReplicaSets are fundamental for deploying and updating applications. By using revision history limits, you can control the footprint of old ReplicaSets. Remember that directly deleting ReplicaSets is discouraged as it can interfere with Kubernetes' rollback and history mechanisms. If you encounter issues with old ReplicaSets not being cleaned up, carefully review your Deployment configuration and investigate potential conflicts within your cluster. Understanding these components and their interactions is crucial for maintaining a healthy and efficient Kubernetes environment.

• Deployment with recreate strategy does not remove old replica set ... | I want to replace an application by running a new deployment. The image tag however does not change, although the image itself might have (that's the reason for the image pull policy always). For t...
• Kubernetes keeping old replicasets with running pods after ... | Oct 29, 2019 ... These answers are taken from this related StackOverflow thread: Clean up "Replica Sets" when updating deployments? . ... kubernetes deployments / ...
• ReplicaSets not removed, even with revisionHistoryLimit set · Issue ... | We've been seeing replicasets going way above the specified .spec.revisionHistoryLimit. $ kubectl get rs --selector=component=resque | wc -l 139 $ kubectl get -ojson deploy resque | jq '.spec.revis...
• Re: [kubernetes-users] Clean up old, empty replica sets created by a ... | (Obviously, I could just delete the replica sets, but I feel like that might be a bad idea if the deployment expects them to stick around.) -- You received this ...
• Kubernetes Deployments in 5 Minutes | by Prakash Waikar | Medium | Deployment controller provides declarative updates for pods.
• Hi there I m noticing that old replica sets aren t being cle Pulumi ... | Hi there I m noticing that old replica sets aren t being cleaned up after a successful deployment I think it kinda makes sense as you d want to be able to roll back to and old one on failure but is th
• Deployment - Unofficial Kubernetes | A Deployment provides declarative updates for Pods and Replica Sets (the next-generation Replication Controller).
• Understanding Deployments and DeploymentConfigs ... | Deployments manage their replica sets automatically, provide declarative updates ... replica sets and scale up the newest one. DeploymentConfig objects can ...
• Redeploy pod when config/secret has been changed/updated : r ... | Posted by u/somzeFiree - 6 votes and 19 comments