# Debugging and Forensics: CSI Mode for Your Cluster #### Kubernetes is great—until something goes wrong. Then, it turns into a black box of cryptic failures, disappearing logs, and misbehaving workloads that refuse to explain themselves. Most debugging attempts follow the same cycle: 1. **Run** `kubectl get pods` and squint at the output. 2. **Try** `kubectl describe pod` and pretend you understand what’s happening. 3. **Start tailing logs, praying for an obvious error message.** 4. **Give up and restart the pod, hoping it magically fixes itself.** But Kubernetes forensics doesn’t have to be an unsolvable crime scene. If you know the right **kubectl** commands, you can **trace, diagnose, and fix problems like a cluster detective**—without resorting to a blind `kubectl delete pod`. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1741540353370/67ec492e-0ba0-48b1-b959-a4ff67fc912a.png align="center") --- ### **1\. Getting Inside a Running Pod: The "SSH Equivalent"** Need to poke around inside a container? `kubectl exec` is your backdoor into the running workload. ```bash kubectl exec -it my-pod -- /bin/sh ``` * `-i` → Keeps the session interactive. * `-t` → Allocates a TTY (so it doesn’t look like garbage). * `-- /bin/sh` → Opens a shell inside the container. If the container is running **Alpine Linux**, it probably needs: ```bash kubectl exec -it my-pod -- /bin/ash ``` For **BusyBox-based** containers: ```bash kubectl exec -it my-pod -- /bin/busybox sh ``` If you don’t know what shell the container has, try this: ```bash kubectl exec -it my-pod -- sh -c 'which bash || which sh || which ash || which busybox' ``` One of them will work. And if your container is **multi-container**, specify which one: ```bash kubectl exec -it my-pod -c my-container -- /bin/sh ``` Now you’re *inside* the running pod, ready to explore. --- ### **2\. Debugging with Logs: Reading the Cluster’s Diary** If a pod is failing, logs are your **first clue**. Instead of checking logs one pod at a time, you can **stream logs across multiple pods at once**: ```bash kubectl logs -f -l app=my-app ``` This pulls logs from **all pods matching the label** `app=my-app`, updating in real-time (`-f` for follow). Need to check logs from a pod that **already crashed**? ```bash kubectl logs my-pod --previous ``` This retrieves logs from the last container instance before it exited. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1741540378420/8d4d447a-1d8e-46b2-bc89-e1ae45ebf7e9.png align="center") --- ### **3\. Finding the Root Cause with** `kubectl describe` If a pod won’t start, `kubectl describe` can reveal **why Kubernetes is mad at you**. ```bash kubectl describe pod my-pod ``` Look for **events at the bottom of the output**. Some common failure messages: * **CrashLoopBackOff** → The container keeps crashing on startup. * **ImagePullBackOff** → Kubernetes can’t pull the image (wrong name or authentication issue). * **ErrImagePull** → Same as above, but the failure happened instantly. * **OOMKilled** → The pod ran out of memory and got terminated. If you see `OOMKilled`, your container probably needs more memory: ```bash kubectl set resources deployment my-app --limits=memory=512Mi ``` This bumps the memory limit, preventing the pod from being **mercilessly executed by the Kubernetes scheduler**. --- ### **4\. Investigating the Cluster’s Health** Pods are just the tip of the iceberg. If **your cluster itself** is struggling, these commands help **diagnose deeper issues**. #### **Checking the Cluster's Overall Health** ```bash kubectl cluster-info ``` This shows whether the API server and core services are running. #### **Inspecting Node Health** ```bash kubectl get nodes kubectl describe node my-node ``` If a node is **NotReady**, it could mean: * The node is out of memory or disk space. * The kubelet process has crashed. * The node has lost network connectivity. Check if the node is running out of resources: ```bash kubectl top node ``` If CPU or memory is maxed out, you may need to **scale up your cluster**. #### **Checking for Failing System Components** ```bash kubectl get componentstatuses ``` This shows the health of **core Kubernetes services** like the scheduler, controller manager, and etcd. --- ### **5\. Debugging a Pod Before It Even Starts** If a pod never even reaches the "Running" state, you can **spin up a temporary debug container** to investigate its environment. ```bash kubectl run debug-shell --rm -it --image=ubuntu-- /bin/sh ``` This gives you a temporary Ubuntu container **inside the same namespace** as your app, letting you inspect networking, DNS resolution, and environment variables **before the real pod launches**. --- ### **6\. Port Forwarding: Expose Internal Services for Debugging** Some services are **only accessible inside the cluster**. If you need to debug a **database or API that’s locked down**, you can use port forwarding: ```bash kubectl port-forward svc/my-service 8080:80 -n my-namespace ``` This maps **port 80 inside the cluster** to **port 8080 on your local machine**. Now you can access the service by visiting [`http://localhost:8080`](http://localhost:8080). Need to connect to a **database inside the cluster**? ```bash kubectl port-forward pod/my-db-pod 5432:5432 -n my-namespace ``` Now you can connect to [`localhost:5432`](http://localhost:5432) as if the database was running locally. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1741540397799/71686352-c074-49a7-912e-1e8bc6088393.png align="center") --- ### **7\. Killing a Stuck Pod the Right Way** Sometimes, a pod refuses to die. If `kubectl delete pod` just sits there doing nothing, try **force deletion**: ```bash kubectl delete pod my-pod --force --grace-period=0 ``` This **bypasses normal termination** and immediately removes the pod from the API server. Still stuck? The issue might be on the **node itself**. Find the node running the pod: ```bash kubectl get pod my-pod -o wide ``` SSH into the node and manually remove the pod’s data: ```bash ssh my-node sudo crictl pods | grep my-pod sudo crictl stopp sudo crictl rmp ``` This **forcefully removes** the pod at the container runtime level. --- ### **Final Thoughts** Debugging Kubernetes **isn’t about guessing**—it’s about **methodically uncovering the truth**. * **Need to inspect a running container?** `kubectl exec` * **Logs disappeared too fast?** `kubectl logs --previous` * **Pod refuses to start?** `kubectl describe pod` * **Cluster acting weird?** `kubectl cluster-info` * **Service unreachable?** `kubectl port-forward` Instead of randomly restarting things and hoping for the best, **use the right kubectl tools to trace the problem to its root cause**. Kubernetes isn’t a mystery. It just **hides its secrets well**—but with the right forensic skills, you can **make your cluster tell you exactly what’s wrong**. Now go forth and debug like a Kubernetes detective. 🚀