Kubernetes PVC Deletion Failed: Troubleshooting Guide

• Introduction
• Step-by-Step Guide
• Code Example
• Additional Notes
• Summary
• Conclusion
• References

Sometimes, you might encounter issues when deleting PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) in Kubernetes, especially if they are in a Terminating state. This guide provides a step-by-step approach to safely and effectively delete PVCs and PVs, addressing potential issues along the way.

1. Verify PVC and PV status:

   kubectl get pvc <pvc-name> -n <namespace>
   kubectl get pv <pv-name>

   Check if they are in Terminating status.

2. Identify any attached resources:

   kubectl get volumeattachment | grep <pv-name>

   If the PV is attached, you need to delete the associated resource (e.g., Pod) first.

3. Delete the Pod using the PVC:

   kubectl delete pod <pod-name> -n <namespace>

   Replace <pod-name> and <namespace> accordingly.

4. Delete the PVC:

   kubectl delete pvc <pvc-name> -n <namespace>

5. Verify PVC deletion:

   kubectl get pvc <pvc-name> -n <namespace>

   The PVC should be removed.

6. Delete the PV (if ReclaimPolicy allows): If the PV's ReclaimPolicy is set to Delete, it should automatically delete. Otherwise:

   kubectl delete pv <pv-name>

7. Address potential issues:

   • PVC stuck in Terminating: Patch the PVC to force finalization:

     kubectl patch pvc <pvc-name> -n <namespace> -p '{"metadata":{"finalizers":null}}'

   • PV stuck in Terminating: Investigate potential issues like volume detachments or underlying storage provider problems.

Note: Replace placeholders like <pvc-name>, <pv-name>, <namespace> with your actual values.

This Python code defines a function that deletes a Kubernetes PersistentVolumeClaim (PVC) and its associated PersistentVolume (PV). It first checks for any attached resources to the PV and raises an error if any are found. It then deletes any Pods using the PVC and proceeds to delete the PVC itself. Finally, it checks the PV's ReclaimPolicy and either deletes the PV automatically or instructs the user to do so manually. The code includes error handling and an example usage scenario.

import kubernetes.client
from kubernetes.client.rest import ApiException

def delete_pvc_and_pv(pvc_name, namespace, pv_name=None):
    """
    Deletes a PersistentVolumeClaim (PVC) and its associated PersistentVolume (PV).

    Args:
        pvc_name (str): Name of the PVC.
        namespace (str): Namespace where the PVC resides.
        pv_name (str, optional): Name of the PV. If not provided, it will be fetched automatically.

    Raises:
        ApiException: If any Kubernetes API call fails.
    """

    # Initialize Kubernetes API client
    kubernetes.config.load_kube_config()
    core_v1_api = kubernetes.client.CoreV1Api()

    try:
        # 1. Verify PVC and PV status
        pvc = core_v1_api.read_namespaced_persistent_volume_claim(pvc_name, namespace)
        print(f"PVC '{pvc_name}' status: {pvc.status.phase}")

        if not pv_name:
            pv_name = pvc.spec.volume_name
        pv = core_v1_api.read_persistent_volume(pv_name)
        print(f"PV '{pv_name}' status: {pv.status.phase}")

        # 2. Identify any attached resources
        volume_attachments = core_v1_api.list_volume_attachment().items
        attached_resources = [va for va in volume_attachments if va.spec.source.persistent_volume_name == pv_name]

        if attached_resources:
            print(f"PV '{pv_name}' is attached to resources:")
            for va in attached_resources:
                print(f"- {va.metadata.name} (kind: {va.spec.source.persistent_volume_name})")
            raise Exception("Cannot delete PV while it's attached to resources. Delete the resources first.")

        # 3. Delete the Pod using the PVC (if applicable)
        # Note: This step assumes the PVC is used by a Pod. Adjust accordingly if not.
        pods = core_v1_api.list_namespaced_pod(namespace, label_selector=f"pvc={pvc_name}").items
        if pods:
            for pod in pods:
                print(f"Deleting Pod '{pod.metadata.name}' using PVC '{pvc_name}'...")
                core_v1_api.delete_namespaced_pod(pod.metadata.name, namespace)

        # 4. Delete the PVC
        print(f"Deleting PVC '{pvc_name}'...")
        core_v1_api.delete_namespaced_persistent_volume_claim(pvc_name, namespace)

        # 5. Verify PVC deletion
        try:
            core_v1_api.read_namespaced_persistent_volume_claim(pvc_name, namespace)
            print(f"PVC '{pvc_name}' still exists.")
        except ApiException as e:
            if e.status == 404:
                print(f"PVC '{pvc_name}' deleted successfully.")
            else:
                raise

        # 6. Delete the PV (if ReclaimPolicy allows)
        if pv.spec.persistent_volume_reclaim_policy == "Delete":
            print(f"PV '{pv_name}' has 'Delete' ReclaimPolicy, it should delete automatically.")
        else:
            print(f"Deleting PV '{pv_name}'...")
            core_v1_api.delete_persistent_volume(pv_name)

    except ApiException as e:
        print(f"Exception when calling Kubernetes API: {e}")

# Example usage:
delete_pvc_and_pv(pvc_name="my-pvc", namespace="my-namespace")

Explanation:

1. Initialization: Imports necessary libraries and initializes the Kubernetes API client.
2. delete_pvc_and_pv function:
   • Takes pvc_name, namespace, and optional pv_name as arguments.
   • Fetches PVC and PV details using the Kubernetes API.
   • Checks for attached resources to the PV. If found, raises an exception to prevent accidental deletion.
   • Deletes any Pods using the PVC (assuming a typical PVC-Pod relationship).
   • Deletes the PVC.
   • Verifies PVC deletion.
   • Deletes the PV if its ReclaimPolicy is set to "Delete". Otherwise, informs the user about manual deletion.
3. Error Handling: Uses try-except blocks to catch ApiException and print informative error messages.
4. Example Usage: Shows how to call the function with specific PVC name and namespace.

Remember to:

• Replace placeholders like my-pvc, my-namespace with your actual values.
• Install the kubernetes Python library: pip install kubernetes
• Configure your Kubernetes credentials (e.g., using kubectl proxy or a kubeconfig file).

General Considerations:

• Order of Deletion: Always delete Pods using the PVC before deleting the PVC itself. This ensures data integrity and prevents potential errors.
• Reclaim Policies: Understand the ReclaimPolicy of your PV. Delete will automatically remove the underlying storage, while Retain requires manual cleanup.
• Namespaces: Specify the correct namespace for both PVCs and Pods. Use kubectl config get-contexts to check your current namespace.
• Data Backup: Before deleting PVCs, especially those with Retain policy, ensure you have a data backup and a plan for restoring it if needed.

Troubleshooting:

• Force Delete PVC: If a PVC is stuck in Terminating due to finalizers, you can use the kubectl patch command to remove them. However, this should be used cautiously as it might lead to data loss.
• PV Detachment Issues: If a PV is stuck in Terminating due to detachment problems, investigate the underlying storage provider logs for errors. You might need to manually detach the volume from the storage provider's side.
• Resource Quotas: If you encounter errors related to resource quotas, check your namespace's limits on PVCs and storage. You might need to adjust them or delete other resources to free up space.

Best Practices:

• Use Labels: Label your PVCs and PVs consistently to easily identify and manage them.
• Automate Deletion: Consider using tools like Helm or Kubernetes Operators to automate the lifecycle management of your PVCs and PVs.
• Monitor Storage Usage: Regularly monitor your storage usage to prevent running out of space and to identify potential issues early on.

Additional Resources:

• Kubernetes Documentation: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• Troubleshooting PersistentVolumes: https://kubernetes.io/docs/tasks/administer-cluster/reclaim-pv/

This guide outlines the steps to safely delete a Kubernetes PV and PVC, addressing potential issues along the way.

Remember: Replace placeholders like <pvc-name>, <pv-name>, and <namespace> with your actual values.

Deleting PersistentVolumes and PersistentVolumeClaims in Kubernetes requires careful consideration to avoid data loss or operational disruptions. This guide provided a step-by-step approach to safely delete these resources, covering verification, resource detachment, and troubleshooting common issues. By following these practices and understanding the underlying concepts, Kubernetes users can confidently manage their storage resources while minimizing risks.

• Can't remove terminating persistent volumes (PV) · Issue #77258 ... | What happened: The pod and namespaces got removed, but sometimes the pv still remains in "Terminating" status even with ReclaimPolicy Delete. Name: pvc-3252ac79-674e-11e9-9651-5404a6f17789 Labels: ...
• How to delete persistent volumes in Kubernetes - Stack Overflow | Aug 7, 2019 ... Delete all the pods, which is using the pvc(you want to delete), then delete the PVC(PersistentVolumeClaim) & PV(PersistentVolume) in sequence.
• pvc stuck in terminating status - Red Hat Learning Community | it probably wont affect me on the exam as i will have enough storage, but it will be good to learn for the future i messed up my pvc command so i tried to --owerwrite it assuming it will just create a new pvc with the changes.however that is not the case once i used oc delete on the first it got stu...
• Kubernetes Cluster stuck on removing PV/PVC - Stack Overflow | Apr 14, 2019 ... Kubernetes: Can't delete PersistentVolumeClaim (pvc) · 18 · How to delete only unmounted PVCs and PVs? 4 · Kubernetes PVC deleting the contents ...
• How To Fix A PVC Stuck in Terminating Status in Kubernetes ... | Having trouble deleting a persistent volume claim (PVC) in Kubernetes? Learn how to patch the PVC to allow the final unmount, allowing you to delete the PVC.
• Kubernetes: How do I delete PV in the correct manner - Stack Overflow | Feb 11, 2019 ... So I deleted the following for es-data: 1) PVC 2) PV They showed as terminating and was left for the weekend. Upon arriving this morning they ...
• PersistentVolumeClaim doesn't bind to a volume when created ... | I created a PersistentVolume and a PersistentVolumeClaim that binds to that volume. Everything worked as expected Went ahead and deleted the claim On creating the same claim again, it refuses to bi...
• Persistent Volumes | Kubernetes | This document describes persistent volumes in Kubernetes. Familiarity with volumes, StorageClasses and VolumeAttributesClasses is suggested. Introduction Managing storage is a distinct problem from managing compute instances. The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, we introduce two new API resources: PersistentVolume and PersistentVolumeClaim. A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes.
• kubernetes - How to increase disk size in a stateful set - Server Fault | Feb 22, 2019 ... I tested this on kubernetes 1.14: kubectl edit pvc for each PVC in the StatefulSet, to increase its capacity. kubectl delete sts -- ...