Kubernetes: View Crashed Pod Logs - Troubleshooting Guide

• Introduction
• Step-by-Step Guide
• Code Example
• Additional Notes
• Summary
• Conclusion
• References

Troubleshooting applications running in Kubernetes often involves analyzing pod logs. When a pod crashes or encounters errors, logs provide valuable insights into the root cause. This guide outlines various methods to effectively view and analyze Kubernetes pod logs using the kubectl logs command.

1. Identify the crashed pod:

   kubectl get pods

   Look for pods with a status of "CrashLoopBackOff" or "Error".

2. View logs of the current/most recent pod instance:

   kubectl logs <pod-name>

   Replace <pod-name> with the actual name of the crashed pod.

3. View logs of previous pod instances:

   kubectl logs <pod-name> --previous

   This command shows logs from the last instance of the pod before it was restarted.

4. View logs for a specific container (if applicable):

   kubectl logs <pod-name> -c <container-name>

   Replace <container-name> with the name of the container within the pod.

5. Follow logs in real-time:

   kubectl logs <pod-name> -f

   This command streams the logs as they are generated, helpful for debugging live issues.

6. Specify a time range for logs:

   kubectl logs <pod-name> --since=1h

   This command shows logs from the last hour. You can adjust the time range as needed (e.g., --since=5m, --since=2d).

7. If the pod name is dynamic (e.g., using a deployment):

   • Use kubectl get pods -o wide to find the pod name based on labels or other identifying information.
   • Alternatively, use kubectl describe deployment <deployment-name> to get details about the deployment and its associated pods. Then, use the pod name from the output to view logs.

This code snippet provides a guide on debugging crashed pods in a Kubernetes cluster. It demonstrates how to identify crashed pods using kubectl commands, view logs of current and previous pod instances, filter logs by container name and time range, and follow logs in real-time. It also includes tips for handling dynamic pod names and suggests using tools like 'stern' for advanced log management.

Let's assume you have a deployment named "my-app" and one of its pods is crashing. Here's how you can debug it:

1. Identify the crashed pod:

kubectl get pods

Output:

NAME                          READY   STATUS             RESTARTS   AGE
my-app-5f4497949c-xyz12      1/1     Running            0          5m
my-app-5f4497949c-abc45      0/1     CrashLoopBackOff   3          1m

Here, my-app-5f4497949c-abc45 is crashing.

2. View logs of the current/most recent pod instance:

kubectl logs my-app-5f4497949c-abc45

3. View logs of previous pod instances:

kubectl logs my-app-5f4497949c-abc45 --previous

4. View logs for a specific container (if applicable):

Assuming the pod has a container named "main-app":

kubectl logs my-app-5f4497949c-abc45 -c main-app

5. Follow logs in real-time:

kubectl logs my-app-5f4497949c-abc45 -f

6. Specify a time range for logs:

kubectl logs my-app-5f4497949c-abc45 --since=30m 

This will show logs from the last 30 minutes.

7. If the pod name is dynamic:

# Get pod name based on labels
kubectl get pods -l app=my-app -o wide

# Get pod name from deployment details
kubectl describe deployment my-app

Once you have the pod name, you can use the commands above to view its logs.

Remember:

• Replace placeholders like <pod-name> and <container-name> with actual values.
• This is a basic example. You can combine these commands and options for more specific debugging scenarios.
• Consider using tools like stern for more advanced log viewing and filtering.

• Understanding Pod Restart Behavior: Pods in a CrashLoopBackOff state are automatically restarted by Kubernetes. Each restart creates a new instance of the pod. The --previous flag is crucial for accessing logs from failed instances to understand why a pod isn't starting successfully.

• Log Storage: Kubernetes itself doesn't persist logs indefinitely. Log storage depends on your cluster configuration:

  • Cluster Logging: Most managed Kubernetes offerings (EKS, GKE, AKS) provide centralized logging solutions.
  • DaemonSet: You can deploy a logging DaemonSet (e.g., using Fluentd or Filebeat) to collect logs from nodes and send them to a central location.
  • EmptyDir: By default, logs are stored in an EmptyDir volume, which is ephemeral. This means logs are lost if the pod is rescheduled to a different node.

• Resource Limits: Be mindful of resource limits set on your pods. If logs exceed the allocated storage, they might be truncated, leading to incomplete debugging information.

• Debugging Strategies:

  • Start Broad, Narrow Down: Begin by examining the most recent logs and work backward through previous instances to identify the point of failure.
  • Look for Errors and Exceptions: Pay close attention to error messages, stack traces, and any unusual patterns in the logs.
  • Correlation: If your application consists of multiple microservices, correlate logs from different pods to trace the flow of requests and pinpoint issues.

• Advanced Tools: While kubectl logs is sufficient for basic log inspection, tools like stern, kubetail, and dedicated log aggregation systems offer features like:

  • Tailing logs from multiple pods simultaneously.
  • Filtering logs based on various criteria (namespace, labels, message content).
  • Real-time log analysis and visualization.

This guide provides a concise summary of commands for investigating crashed pods in Kubernetes.

Note: Replace placeholders like <pod-name> and <container-name> with actual names.

Mastering the art of viewing and analyzing pod logs is an essential skill for anyone working with Kubernetes. By understanding the nuances of kubectl logs and leveraging its various options, you can effectively troubleshoot application issues, identify root causes, and ensure the smooth operation of your Kubernetes deployments. Remember to establish a robust logging strategy for your cluster to prevent log loss and facilitate easier debugging in the future. As you become more comfortable with basic log inspection, explore advanced logging tools and techniques to streamline your troubleshooting workflows and gain deeper insights into your application's behavior within the Kubernetes environment.

• How to check the logs of running and crashed pods in Kubernetes ... | If you have managed any kind of Linux bases servers, you have probably used commands like cat and tail to check your server logs. Here I will show yo…
• kubernetes - How to view log of crash pod of a deployment - Stack ... | Jun 7, 2020 ... If a Deployment-managed Pod crashes, the same Pod will restart, and you can look at its logs using kubectl logs --previous the same as ...
• How to see logs of crashed pods in a kubernetes cluster - Server Fault | Jun 5, 2024 ... I tried using this command kubectl logs --previous but as the name of the pod is dynamic, I cannot see the logs at all with --previous ...
• Get Kubernetes Logs With kubectl | Warp | Learn how to get the logs of pods, containers, deployments, and services in Kubernetes using the kubectl command. Troubleshoot a cluster stuck in CrashloopBackoff, ImagePullBackoff, or Pending error states.
• CrashLoopBackOff - how to get logs from the first pod that failed and ... | Posted by u/InquisitiveProgramme - 24 votes and 20 comments
• Kubernetes See Logs Of The Crashed Pod using Kubectl | by Baris ... | Normally, we see only the current pod’s log when we look through the logs.
• K8s Troubleshooting — Check Previous Crashed Pod Logs | by ... | K8s Troubleshooting handbook
• Why do my pods keep crashing? - Discuss Kubernetes | I’ve checked logs and described pods and the only thing of note is. Pod sandbox changed, it will be killed and re-created. I’m thinking that there may be a misconfiguration. I’ve shown versions first so maybe someone can confirm that they are ok. Cluster information: Kubernetes version: 1.29 Cloud being used: AWS Installation method: terraform Host OS: RHEL9 CNI and version: flannel v0.24.0 CRI and version: containerd 1.7.11 [ec2-user@vagrant-tf-master01 ~]$ rpm -qa | grep kube kub...
• kubectl logs: How to Get Pod Logs in Kubernetes (With Examples) | Pod logs play a critical role in troubleshooting containerized applications. Click to learn how to use the "kubectl logs" command to get Pod logs.