Kubernetes Troubleshooting in 2026: 10 Pod Errors, Their Real Causes and the Fix for Each

It is 1:50 on a Tuesday morning and you are the only person awake with cluster access. The deployment went out at 11 PM, the dashboard is red, and kubectl get pods shows eight replicas at 0/1 with a restart count climbing every few minutes. The app worked on your laptop. What you do in the next four minutes decides whether this is a twenty minute incident or a three hour one, and the difference is almost never cleverness. It is reading the right output in the right order.

The Kubernetes Troubleshooting Loop: Four Commands Before You Touch Any YAML

Every Pod failure answers one question first: did the container ever run? Scheduling problems, image problems and runtime problems look identical on a dashboard and completely different in kubectl. These four commands, in this order, separate them in under a minute.

# 1. Which Pods are unhealthy, and on which node?
kubectl get pods -n payments -o wide

# 2. The Events tail at the bottom is the real answer most of the time
kubectl describe pod checkout-7d9f8c6b4-xk2lp -n payments

# 3. Logs from the container that already died, not the one restarting now
kubectl logs checkout-7d9f8c6b4-xk2lp -n payments --previous --tail=50

# 4. Namespace-wide view, newest last
kubectl get events -n payments --sort-by=.lastTimestamp | tail -20

Read Events before logs. Logs only exist if a process started and wrote something; Events exist even when the container never got as far as pulling an image. Events are garbage collected after a while, so the Pod that broke at 2 AM may show nothing at 9 AM. Capture describe output before you delete anything.

Read the Event source too, because it narrows the fix immediately. default-scheduler means no node would take the Pod, so your application code is irrelevant. kubelet with Failed to pull image points at the registry. kubelet with Liveness probe failed means your app is running and something you configured is killing it.

Kubernetes troubleshooting table: 10 pod errors, symptom, cause and first fix

Keep this open in a second tab during an incident. The third column is what you look for in kubectl output, which is the part most reference pages leave out.

Errors 1 to 3: CrashLoopBackOff and the Backoff Clock Nobody Explains

CrashLoopBackOff is not a diagnosis. It is the kubelet telling you that it has restarted a container, the container died again, and it is now waiting before the next attempt. The Kubernetes Pod lifecycle documentation describes that wait as an exponential back-off starting at 10 seconds, doubling on each failure, capped at 300 seconds. That cap is why your Pod feels frozen: after six or seven failures you are watching a restart once every five minutes, and a small log change you push will take that long to show up.

Two changes to that clock are worth knowing, because your version decides which you get. The alpha feature gate ReduceDefaultCrashLoopBackOffDecay, available since Kubernetes v1.33 and off by default, starts retries at 1 second and caps them at 60. Separately, KEP-5593 added a kubelet field, crashloopbackoff.maxContainerRestartPeriod, documented as beta and enabled by default in v1.35, accepting 1 to 300 seconds. If developers complain about slow feedback, set the kubelet field rather than enabling the alpha gate.

Telling the three crash loops apart

The first command is always the same, and the flag matters: without --previous you get logs from the container starting now, which are usually empty.

# The dead container's final words
kubectl logs deploy/checkout --previous --tail=50 -n payments

# The exact termination record, including exit code and reason
kubectl get pod checkout-7d9f8c6b4-xk2lp -n payments \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

Now read the exit code. A 1 or 2 with a stack trace is your application refusing its own configuration, which is error 1. An empty log plus executable file not found in Events is error 2, and in 2026 that usually means you built on an Apple Silicon laptop and deployed to amd64 nodes. A healthy startup sequence that simply stops, with Liveness probe failed in Events, is error 3: the app is fine and the probe is wrong.

Most crash loops that are not config errors trace back to how the image was built. Also read: Docker Best Practices in 2026: 10 Mistakes That Bloat Your Images and How to Fix Each One, which covers the build-side causes in the same symptom and fix format. Container operations across AWS and Kubernetes form a graded module in 360DT's AWS Solutions Architect and DevOps Engineer course, where you break a deployment on purpose and diagnose it live.

Error 4: ImagePullBackOff, and Why Docker Hub Quotas Trigger It

ImagePullBackOff has a sibling you should learn to distinguish. ErrImagePull is the first failed attempt; ImagePullBackOff is the kubelet waiting before trying again. If you catch the Pod early you see the first, and the error text is more specific. Four causes produce the identical status, and the Events tail is the only thing that separates them:

• A tag that does not exist. Usually a typo, or a CI job that pushed v1.4.2 while your manifest asks for 1.4.2. Events say manifest unknown.
• A private registry with no credentials. Events say unauthorized or authentication required. The Pod needs an imagePullSecrets entry, or the node needs an IAM role that can read the registry.
• A quota. Events contain toomanyrequests. Docker's pull usage documentation states the limit is calculated on a 6 hour basis: 100 pulls per IPv4 address or IPv6 /64 subnet unauthenticated, 200 for a free authenticated personal account, unlimited on paid plans. Twenty nodes behind one NAT gateway burn through 100 pulls during a single rollout.
• An architecture mismatch. Events say no match for platform in manifest. Graviton and other arm64 node pools have moved this one from rare to weekly.

For a private Docker Hub repository, use a scoped access token rather than your account password:

kubectl create secret docker-registry regcred \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=<username> \
  --docker-password=<access-token> \
  -n payments

# Then reference it in the Pod spec
#   imagePullSecrets:
#   - name: regcred

The structural fix is to stop pulling public images into production. Mirror the handful you depend on into Amazon ECR, Azure Container Registry or your own Harbor, and pin them by digest rather than tag, so a rollout never depends on a third party's quota on your worst day. Registry design, IAM for pull access and multi-architecture builds sit in the DOP-C02 half of a live AWS DevOps engineering program, with the ACR and AKS equivalents in the Azure Solutions Architect and DevOps Engineer course.

Error 5: OOMKilled, Exit Code 137 and the Heap You Forgot to Cap

This failure wastes the most time, because your application logs look perfect. No exception, no shutdown message, no last line of warning. The kernel's OOM killer sent SIGKILL when the container crossed its memory limit, and SIGKILL cannot be caught or handled. Exit code 137 is 128 plus signal 9.

kubectl describe pod checkout-7d9f8c6b4-xk2lp -n payments | grep -A5 "Last State"

#     Last State:     Terminated
#       Reason:       OOMKilled
#       Exit Code:    137
#       Started:      Tue, 30 Oct 2026 01:48:11 +0530
#       Finished:     Tue, 30 Oct 2026 01:52:40 +0530

Raising the limit is the obvious move and often the wrong one. In JVM, Node.js and Python services the usual cause is a runtime sizing its own heap from what it believes the machine has rather than from the cgroup limit. A Spring Boot service with a 512Mi limit and no -XX:MaxRAMPercentage will cheerfully try to use more than the container allows. Cap the runtime first, then set the container limit about 25 percent above that cap to cover thread stacks and non-heap overhead.

Kubernetes v1.35 promoted in-place Pod resize to general availability, so you no longer have to recreate a Pod to change its resources. A plain patch on the Pod spec is still rejected; the change goes through the resize subresource, and your kubectl client must be v1.32 or newer for the flag to exist.

kubectl patch pod checkout-7d9f8c6b4-xk2lp -n payments --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"limits":{"memory":"1Gi"},"requests":{"memory":"1Gi"}}}]}}'

Whether the container restarts depends on its resizePolicy: CPU changes apply without interrupting the process, while memory behaviour varies by version, so check the resize documentation before promising anyone a restart-free change. In an incident this buys breathing room on one Pod while you fix the Deployment properly.

Errors 6 to 8: Pods Stuck in Pending or ContainerCreating

A Pending Pod has not been assigned to a node, so your application has not been consulted at all. The scheduler writes its reason into Events in an unusually readable form, for example 0/6 nodes are available: 3 Insufficient cpu, 3 node(s) had untolerated taint. Read the counts: they tell you whether this is a capacity problem or a policy problem.

The trap that catches people new to clusters is that a node advertised as 2 vCPU does not offer 2000m to your Pods. The kubelet reserves capacity for itself, so allocatable is always lower than capacity, and a Pod requesting a flat cpu: 2 will never schedule on it.

# Allocatable capacity and what is already committed, per node
kubectl describe node ip-10-0-3-41.ap-south-1.compute.internal | grep -A12 "Allocated resources"

# Volumes waiting to bind
kubectl get pvc -n payments

If the Pod moves to ContainerCreating and stays there, the scheduler has done its job and the kubelet is stuck: a missing Secret mounted as a volume, a CSI driver that cannot attach a disk in the right availability zone, or a sandbox the CNI plugin could not wire up. A PVC sitting in Pending under a WaitForFirstConsumer StorageClass is normal until a Pod needs it, so check in that order before panicking about storage.

Probe Failures That Restart a Perfectly Healthy Pod

Here is a position worth defending: do not add a liveness probe unless you can name the specific hang it is meant to catch. A readiness probe is almost always right, because it only removes a Pod from the Service endpoints. A liveness probe kills your process. If you cannot describe the deadlock it rescues you from, you have built a restart generator and called it reliability.

The defaults are the second trap. They suit small, fast handlers and they are unforgiving of a service doing real work on a busy node.

A slow-starting service, classically a JVM application loading a model or warming a cache, should use a startupProbe with a generous failureThreshold. While a startup probe runs, the liveness probe is held off entirely: patient at boot, strict afterwards. Setting initialDelaySeconds to 120 to achieve the same thing penalises every restart for the life of the Deployment.

One more mistake that reads as a mystery: a liveness endpoint that checks the database. The database blips for thirty seconds, every Pod fails liveness, Kubernetes restarts the entire fleet at once, and the restart storm keeps the database busy. A liveness endpoint should answer one question only, which is whether this process can still serve. Dependency health belongs in readiness, or in monitoring.

Errors 9 and 10: Config Keys That Do Not Exist, and Service DNS That Does Not Resolve

CreateContainerConfigError is the friendliest error in Kubernetes, because Events name the exact problem: couldn't find key DB_PASSWORD in Secret payments/checkout-secrets. The Secret exists, the Pod found it, and the key inside is spelled differently from what the manifest asks for. Base64 is not involved and the value is not wrong. Only the key name is.

Error 10 survives into the afternoon, because the Pod is Running and nothing looks broken. Service to service calls time out, and the cause is almost always the Service name, the namespace or the port. In-cluster DNS resolves <service>.<namespace>.svc.cluster.local, so a Pod in default calling a bare checkout that lives in payments resolves nothing. The port confusion is subtler: a Service's port is what callers dial, while targetPort is the container port traffic lands on. Reverse those and you get a Service with healthy endpoints that answers nothing.

# Does the Service have any endpoints at all? Empty output means the selector matches no ready Pod
kubectl get endpointslices -n payments -l kubernetes.io/service-name=checkout

# Compare the Service selector against the Pod labels
kubectl get svc checkout -n payments -o jsonpath='{.spec.selector}'
kubectl get pods -n payments --show-labels

An empty endpoint list with healthy Pods means the Service selector and the Pod labels disagree, which is a copy and paste error rather than a networking fault. If endpoints exist and traffic still fails, suspect a NetworkPolicy. Policy debugging is a standing exercise in a live AKS and Azure DevOps program, because it teaches the difference between a Pod that is running and a Pod that is reachable.

Kubernetes Troubleshooting With kubectl debug When There Is No Shell

Modern base images are why kubectl exec fails more often than it used to. A distroless or scratch image holds your binary and little else, so there is no /bin/sh to exec into and no curl, ps or nslookup to run. Ephemeral containers, stable since Kubernetes v1.25, attach a second container into the running Pod's namespaces.

# Attach a toolbox into the running Pod, sharing the app container's process namespace
kubectl debug -it checkout-7d9f8c6b4-xk2lp -n payments \
  --image=busybox:1.36 --target=app --profile=general

# Check Service DNS from inside the cluster's network
nslookup checkout.payments.svc.cluster.local

# A root shell on the node itself, for kubelet logs and disk pressure
kubectl debug node/ip-10-0-3-41.ap-south-1.compute.internal -it --image=ubuntu

Pass --profile explicitly: the documentation marks the legacy profile deprecated and recommends naming one such as general, while node-level work like packet capture needs sysadmin. That node shell is the command most people do not know exists, and it is what you want when several Pods on one node misbehave and the rest are fine. Linux fluency decides how far you get once you are in there. Also read: Linux Commands Cheat Sheet for DevOps in 2026.

Honest caveat, since this is where enthusiasm usually outruns judgement. If you run one web application and one database with a team of three, Kubernetes is probably the wrong platform, and no amount of troubleshooting skill changes that arithmetic. Amazon ECS on Fargate, Azure Container Apps or a plain managed VM group will cost you fewer 2 AM hours. Kubernetes earns its complexity at many services, many teams and real traffic variation. Learn it because the jobs are there and the concepts transfer, not because it suits every workload.

A Worked Example: One Broken Deployment, Diagnosed End to End

Back to the 1:50 AM incident. Picture a two-person platform team at a mid-size insurer in Pune running a quotes API on a six-node managed cluster. The 11 PM release added a payment provider, which meant a new secret and a slightly heavier process. Here is the whole session, with the reasoning at each step.

Step 1. kubectl get pods -n payments -o wide shows eight Pods at 0/1, status CreateContainerConfigError, restarts 0. Restarts at zero is the clue: nothing crashed, because nothing started. Config family, not application family, so the logs are empty and there is no point opening them.

Step 2. kubectl describe pod ends with Error: couldn't find key PROVIDER_API_KEY in Secret payments/checkout-secrets. The manifest asks for PROVIDER_API_KEY; the pipeline stored provider_api_key, lowercase, because whoever wrote that step copied the provider's own naming. Two minutes in, and the fix is one line in the key field.

Step 3. After the fix the Pods reach Running, then CrashLoopBackOff with restarts climbing. Different family, different command. kubectl logs deploy/checkout --previous --tail=50 shows a clean startup, three log lines, then nothing. No exception. That absence is the signature of the platform killing the process rather than the process failing.

Step 4. kubectl describe pod | grep -A5 "Last State" returns Reason: OOMKilled, Exit Code: 137. The limit is 256Mi, unchanged for a year, and the new provider client pulled in a dependency that enlarged the startup footprint. Nobody reads memory footprints in a dependency bump.

Step 5. Mitigation: raise the memory limit to 512Mi and roll out. Pods go Ready in under a minute, and service is back at roughly 02:10. The real fix lands the next working day: cap the runtime heap so it respects the cgroup, keep the container limit about 25 percent above that cap, add a startupProbe so the slower boot does not trip liveness at the next deploy, and add the secret key name to the pipeline's validation step, or the same mismatch returns at the next integration.

Notice what diagnosed this. Not a dashboard, not a tracing tool, and no knowledge of the application code. Two status values, one Events line, one Last State block. That is the whole skill, and it is why cluster debugging transfers across companies and stacks. Teams running model-serving workloads hit the same failures with GPU nodes and much larger images, which is why container debugging sits alongside deployment in the MLOps Engineer course and in the serving module of the AI Engineer course. The full certifications overview lays out the sequence if you are deciding what to aim at first.

Before committing to any program, sit in on a free webinar or a demo class and watch a cluster get broken and repaired live. That tells you more about whether this work suits you than any syllabus. Also read: DevOps Engineer Salary in India 2026 for what the certification actually moves in pay.

What I would do in your position

Build the triage reflex before you build breadth. Spend one weekend breaking a cluster deliberately: a bad tag, a wrong secret key, a 64Mi limit on a JVM app, a liveness probe with a one second timeout, a Service whose selector matches nothing. Five deliberate failures, five rounds of describe and logs, and you will recognise every one of them on sight afterwards. That beats reading another overview of control plane components, and it is the difference between a twenty minute incident and a three hour one.

Then pick one managed platform and certify on it, because the troubleshooting concepts are portable while the surrounding plumbing is not. For roles in Indian GCCs and product companies, AWS still carries the broadest demand, and the AWS Solutions Architect and DevOps Engineer course runs SAA-C03 and DOP-C02 together over 12 weeks of live weekend sessions, which matches how these roles are hired. Start with a free webinar if you want to test the format first.

Related guides

• Kubernetes Tutorial for Beginners 2026: Deploy Your First App With kubectl in 8 Steps build the deployment you are going to break here, step by step.
• Docker vs Kubernetes in 2026: Real Costs, 14 Criteria and Which to Learn First in India decide whether you need a cluster at all before you learn to debug one.
• DevOps Interview Questions 2026: 15 Real Questions, Model Answers and the Red Flags to Avoid the crash loop and OOMKilled questions come up in almost every round.
• Site Reliability Engineer Salary in India 2026: Advertised Pay by Experience, Company Type and the Real SRE Premium what the on-call skill is worth once you are good at it.
• Cloud Engineer Jobs in Mumbai 2026: Pay Bands, the Data Centre Shift and Who Is Hiring where cluster operations roles are concentrated in western India.

Frequently asked questions

About this guide. 360 Digital Transformation is an Anthropic Partner and a Microsoft Authorized Training Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Tools and versions change quickly; commands and figures cited were checked on 30 October 2026.