Kubernetes Pod Resource Monitoring: CPU & Memory

• Introduction
• Step-by-Step Guide
• Code Example
• Additional Notes
• Summary
• Conclusion
• References

Troubleshooting high CPU usage in Kubernetes pods requires a multi-faceted approach. Here are some techniques to help you identify and resolve the root cause:

1. Use kubectl top: This command shows resource usage for pods.

   kubectl top pod <pod-name>

2. Check cgroups: Kubernetes uses cgroups to limit resource usage. You can inspect these directly.

   cat /sys/fs/cgroup/cpu/kubepods/*/pod*/cpuacct.usage_percpu

3. Install monitoring tools: For more detailed and historical data, use tools like Prometheus or Datadog. These provide dashboards and alerting.
4. Define resource requests and limits: When deploying pods, specify how much CPU and memory they can request and use. This helps with scheduling and prevents resource starvation.

   resources:
     requests:
       memory: "64Mi"
       cpu: "100m"
     limits:
       memory: "128Mi"
       cpu: "250m"

5. Remember metrics delay: There's a slight delay in metrics gathering, so newly created pods might not show usage immediately.

This guide provides code examples for monitoring CPU usage of Kubernetes pods. You can use kubectl top for real-time views, inspect cgroups for detailed usage per core, or install monitoring tools like Prometheus. Defining resource requests and limits in your pod definition YAML helps manage resource allocation. Remember that there might be a delay in metrics gathering for newly created pods.

This guide provides code examples for monitoring CPU usage of your Kubernetes pods using various methods.

1. Using kubectl top

This command provides a real-time view of CPU and memory usage for a specific pod.

# Replace <pod-name> with the actual pod name
kubectl top pod <pod-name> 

2. Inspecting cgroups

Kubernetes utilizes cgroups to enforce resource limits. You can directly access these cgroups to view CPU usage.

# This command lists CPU usage per CPU core for all pods
cat /sys/fs/cgroup/cpu/kubepods/*/pod*/cpuacct.usage_percpu

# To get the total CPU usage for a specific pod, you can use the following command:
# Replace <pod-uid> with the actual pod UID
cat /sys/fs/cgroup/cpu/kubepods/*/pod<pod-uid>/cpuacct.usage 

3. Installing Monitoring Tools (Prometheus Example)

Prometheus is a popular open-source monitoring system. Here's how to install the Prometheus Operator using Helm:

# Add the Prometheus Operator Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Update your local Helm chart repository cache
helm repo update

# Install the Prometheus Operator
helm install prometheus prometheus-community/kube-prometheus-stack

4. Defining Resource Requests and Limits in YAML

When deploying your applications, specify resource requests and limits in your pod definition YAML file. This helps Kubernetes schedule pods efficiently and prevents resource starvation.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: my-app-image:latest
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "250m"

5. Remembering Metrics Delay

Keep in mind that there might be a slight delay in metrics gathering. Newly created pods might not immediately show resource usage data.

These examples provide a starting point for monitoring CPU usage in your Kubernetes cluster. Remember to adapt these examples to your specific needs and environment.

• Check for resource contention on the node: High CPU usage in a pod might be a symptom of overall node saturation. Use kubectl top nodes to investigate.
• Profile your application: Tools like kubectl exec -it <pod-name> -- top or dedicated profilers can pinpoint code hotspots within the container.
• Consider vertical scaling: If a pod consistently hits CPU limits, increasing its limit (and potentially the node size) might be necessary.
• Look for log errors or warnings: Application-level issues can manifest as high CPU usage. Check pod logs for clues.
• Restart the pod: Sometimes a temporary glitch can cause high CPU; a restart might resolve it.
• Use resource quotas: At the namespace level, quotas prevent resource hogging and ensure fairness.
• Monitor over time: A single point-in-time measurement isn't enough. Use monitoring tools to establish baselines and identify trends.
• Check for Kubernetes events: Events can reveal scheduling failures, resource pressure, or other relevant issues. Use kubectl get events.

This document summarizes various methods to check the CPU usage of Kubernetes pods:

Important Note: Metrics gathering has a slight delay, so newly created pods might not show usage immediately.

Understanding how to monitor and manage CPU usage is crucial for maintaining a healthy and efficient Kubernetes cluster. By employing the techniques and tools outlined in this guide, developers and administrators can gain valuable insights into their application's resource consumption, identify bottlenecks, and ensure optimal performance. Remember to establish clear monitoring practices, define appropriate resource limits, and leverage the troubleshooting steps provided to address high CPU usage effectively. By staying proactive and informed, you can ensure your Kubernetes applications run smoothly and efficiently.

• Checking Kubernetes pod CPU and memory - Octopus Deploy | Learn how to check a pod's resource usage in Kubernetes.
• Checking kubernetes pod CPU and memory without using third ... | Apr 7, 2021 ... It looks like you want to check memory and CPU usage of the Pods that are Completed . I tried the k9s tool and I also cannot see CPU, ...
• Kubectl Top Pod/Node - Monitor K8s Resource Usage | SigNoz | Learn how to use kubectl top to monitor resource usage in Kubernetes clusters. Master node and pod monitoring for optimal performance.
• kubectl top pod | Kubernetes | Synopsis Display resource (CPU/memory) usage of pods. The 'top pod' command allows you to see the resource consumption of pods. Due to the metrics pipeline delay, they may be unavailable for a few minutes since pod creation. kubectl top pod [NAME | -l label] Examples # Show metrics for all pods in the default namespace kubectl top pod # Show metrics for all pods in the given namespace kubectl top pod --namespace=NAMESPACE # Show metrics for a given pod and its containers kubectl top pod POD_NAME --containers # Show metrics for the pods defined by label name=myLabel kubectl top pod -l name=myLabel Options -A, --all-namespaces If present, list the requested object(s) across all namespaces.
• How do you monitor CPU/Memory usage of pod with Prometheus ... | Posted by u/hksparrowboy - 6 votes and 3 comments
• linux - Kubernetes - Find per core statistics for the pod - Server Fault | Jan 8, 2024 ... kubectl command gets resource utilization metrics from cAdvisor via the metrics-server. This command requires Metrics Server to be correctly ...
• Resource Management for Pods and Containers | Kubernetes | When you specify a Pod, you can optionally specify how much of each resource a container needs. The most common resources to specify are CPU and memory (RAM); there are others. When you specify the resource request for containers in a Pod, the kube-scheduler uses this information to decide which node to place the Pod on. When you specify a resource limit for a container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set.
• Monitoring Kubernetes Pod Performance Metrics | Datadog | Learn about the key Kubernetes metrics that can help you track your orchestrated, containerized infrastructure.
• Checking Kubernetes pod CPU and memory utilization | by Harold ... | To check the CPU and memory utilization of Kubernetes pods, you can use various tools and commands within Kubernetes and external…