🐶
Kubernetes

Kubernetes Namespace Stuck in Terminating: How to Fix

By Jan on 01/15/2025

Learn how to troubleshoot and resolve a Kubernetes Namespace stuck in "Terminating" status, preventing its deletion and resource cleanup.

Kubernetes Namespace Stuck in Terminating: How to Fix

Table of Contents

Introduction

Deleting a Kubernetes namespace can sometimes get stuck in a "Terminating" state. This usually happens when the namespace contains resources that are still being cleaned up or when there are underlying issues preventing the deletion. This guide provides a step-by-step approach to troubleshoot and resolve such scenarios, allowing you to successfully remove the stuck namespace.

Step-by-Step Guide

  1. Identify the stuck namespace:

    kubectl get ns --field-selector status.phase=Terminating

  2. Check for remaining resources:

    kubectl get all -n <namespace-name>

  3. Force delete resources (if any):

    kubectl delete <resource-type> <resource-name> -n <namespace-name> --grace-period=0 --force

  4. Inspect namespace details:

    kubectl get namespace <namespace-name> -o yaml

  5. Edit the namespace and remove finalizers:

    kubectl edit namespace <namespace-name>
    • Remove the kubernetes finalizer from the spec.finalizers list.

  6. Save the changes and verify deletion:

    kubectl get ns

Note: If the issue persists, consider checking for:

  • Third-party resources: Some resources might require manual cleanup.
  • Controller issues: Check controller logs for errors.
  • Cluster issues: In rare cases, cluster-level problems might cause this.

Code Example

The Python code checks for and attempts to resolve a stuck Kubernetes namespace. It first identifies if the specified namespace is in a 'Terminating' state. If so, it lists any remaining resources within the namespace. The script includes a commented-out section for force-deleting these resources. It then retrieves and displays the namespace details, attempts to remove the 'kubernetes' finalizer from the namespace, and finally verifies if the namespace has been successfully deleted.

import kubernetes

# Kubernetes configuration
kubernetes.config.load_kube_config()

# Namespace name
namespace_name = "your-namespace-name"

# 1. Identify the stuck namespace
try:
    namespaces = kubernetes.client.CoreV1Api().list_namespace(field_selector="status.phase=Terminating").items
    stuck_namespaces = [ns.metadata.name for ns in namespaces if ns.metadata.name == namespace_name]
    if stuck_namespaces:
        print(f"Stuck namespace found: {stuck_namespaces[0]}")
    else:
        print(f"Namespace '{namespace_name}' is not stuck in Terminating state.")
        exit()
except kubernetes.client.exceptions.ApiException as e:
    print(f"Error: {e}")
    exit()

# 2. Check for remaining resources
try:
    resources = kubernetes.client.CoreV1Api().list_namespaced_resource("all", namespace_name).items
    if resources:
        print(f"Resources found in namespace '{namespace_name}':")
        for resource in resources:
            print(f"- {resource.kind}/{resource.metadata.name}")
    else:
        print(f"No resources found in namespace '{namespace_name}'.")
except kubernetes.client.exceptions.ApiException as e:
    print(f"Error: {e}")
    exit()

# 3. Force delete resources (if any) - uncomment and modify if needed
# for resource in resources:
#     try:
#         kubernetes.client.CoreV1Api().delete_namespaced_resource(
#             resource.kind.lower(), namespace_name, resource.metadata.name, grace_period_seconds=0, propagation_policy="Foreground"
#         )
#         print(f"Force deleted {resource.kind}/{resource.metadata.name} in namespace '{namespace_name}'.")
#     except kubernetes.client.exceptions.ApiException as e:
#         print(f"Error deleting {resource.kind}/{resource.metadata.name}: {e}")

# 4. Inspect namespace details
try:
    namespace = kubernetes.client.CoreV1Api().read_namespace(namespace_name)
    print(f"Namespace details:\n{namespace}")
except kubernetes.client.exceptions.ApiException as e:
    print(f"Error: {e}")
    exit()

# 5. Edit the namespace and remove finalizers
try:
    # Remove 'kubernetes' finalizer
    namespace.spec.finalizers = [f for f in namespace.spec.finalizers if f != "kubernetes"]
    kubernetes.client.CoreV1Api().patch_namespace(namespace_name, namespace)
    print(f"Removed 'kubernetes' finalizer from namespace '{namespace_name}'.")
except kubernetes.client.exceptions.ApiException as e:
    print(f"Error: {e}")
    exit()

# 6. Verify deletion
try:
    namespaces = kubernetes.client.CoreV1Api().list_namespace().items
    existing_namespaces = [ns.metadata.name for ns in namespaces]
    if namespace_name not in existing_namespaces:
        print(f"Namespace '{namespace_name}' deleted successfully.")
    else:
        print(f"Namespace '{namespace_name}' is still present.")
except kubernetes.client.exceptions.ApiException as e:
    print(f"Error: {e}")
    exit()

Before running the code:

  • Install the Kubernetes Python client: pip install kubernetes
  • Configure kubectl: Make sure your kubectl is configured to access your cluster.
  • Replace placeholders:
    • Replace "your-namespace-name" with the actual name of the stuck namespace.
  • Use caution with force deletion: Force deleting resources can lead to data loss. Only uncomment and use section 3 if you are sure you want to delete all remaining resources in the namespace.

This code provides a more automated approach to resolving stuck namespaces compared to manual command execution. Remember to adapt it to your specific needs and use it responsibly.

Additional Notes

  • Understanding Finalizers: Finalizers are mechanisms in Kubernetes that allow custom controllers to perform cleanup actions before an object (like a namespace) is deleted. The "kubernetes" finalizer is used by the Kubernetes garbage collector itself.
  • Force Deletion Caution: Use --grace-period=0 and --force with extreme caution. It bypasses the standard grace period and can lead to data inconsistency or orphaned resources.
  • Alternative to Editing Namespace: Instead of manually editing the namespace YAML, you can use kubectl patch to remove the finalizer: 
    kubectl patch namespace <namespace-name> -p '{"metadata":{"finalizers":null}}'
  • Common Culprits: Namespaces often get stuck due to:
    • PersistentVolumes: Ensure PVs are not bound to PVCs in the namespace.
    • Custom Resource Definitions (CRDs): Resources created by CRDs might have their own finalizers and cleanup procedures.
  • Debugging:
    • Describe Resources: Use kubectl describe on stuck resources for more information.
    • Events: Check events related to the namespace and its resources: kubectl get events -n <namespace-name>
  • Prevention:
    • Clean Up Before Deletion: Delete dependent resources (Deployments, StatefulSets, etc.) before deleting the namespace.
    • Resource Tracking: Use labels to easily identify and manage resources within a namespace.
  • Cluster-Level Issues: If you suspect a cluster-wide problem, consult your cloud provider's documentation or seek support from your Kubernetes administrator.

Summary

This guide provides steps to troubleshoot and delete a Kubernetes namespace stuck in the "Terminating" state.

Steps:

  1. Identify the stuck namespace: Use kubectl get ns --field-selector status.phase=Terminating to list namespaces in this state.

  2. Check for remaining resources: Execute kubectl get all -n <namespace-name> to see any resources still present within the namespace.

  3. Force delete resources (if any): Remove remaining resources with kubectl delete <resource-type> <resource-name> -n <namespace-name> --grace-period=0 --force.

  4. Inspect namespace details: Use kubectl get namespace <namespace-name> -o yaml to examine the namespace configuration.

  5. Edit the namespace and remove finalizers: Open the namespace for editing with kubectl edit namespace <namespace-name> and remove the kubernetes finalizer from the spec.finalizers list.

  6. Save the changes and verify deletion: Save the edited namespace and confirm its deletion with kubectl get ns.

Troubleshooting:

If the namespace remains stuck, investigate:

  • Third-party resources: Manually clean up resources not managed by Kubernetes.
  • Controller issues: Review controller logs for errors related to the namespace.
  • Cluster issues: Explore potential cluster-wide problems affecting namespace deletion.

Conclusion

By following these steps, you can gain a better understanding of why a namespace is stuck and take the necessary steps to remove it. Remember to proceed with caution, especially when using force deletion, and always double-check for any dependencies or potential issues before taking action. This approach helps ensure a cleaner and more manageable Kubernetes environment.

References

Were You Able to Follow the Instructions?

😍Love it!
😊Yes
😐Meh-gical
😞No
🤮Clickbait

Related posts

Kubernetes Namespace Switching: A Quick Guide

Learn how to easily switch between namespaces in Kubernetes and streamline your workflow for managing resources in different environments.

Kubernetes Service port vs targetPort: What's the difference?

This article explains the difference between targetPort and port in Kubernetes Service definition and when to use each.

Kubernetes External IP Pending: Troubleshooting Guide

Troubleshoot and resolve the frustrating "External IP Pending" status for your Kubernetes services with this comprehensive guide.