Skip to content

Debugging Applications

Kubernetes provides tools to help troubleshoot and debug problems with applications.

Debugging usually means understanding how the objects interact with each other (Deployment → ReplicaSet → Pod, Service → EndpointSlice → Pod), checking the status and events of each object, and finally checking the logs for any last clues.

Resources

  • Troubleshooting

    Inspect pod status, events and logs, and start a debug shell with oc debug.

    Learn more

  • Debugging Applications

    Read about how to debug applications that are deployed into Kubernetes and not behaving correctly.

    Learn more

  • Debugging Services

    You've run your Pods through a Deployment and created a Service, but you get no response when you try to access it. What do you do?

    Learn more

  • Debugging Running Pods

    Use kubectl debug and ephemeral containers to troubleshoot pods that have no shell or debugging tools.

    Learn more

References

A broken application to practice on

Create a project and deploy an application that has several problems:

oc new-project debug

oc apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: app
          image: quay.io/nginx/nginx-unprivileged:1.290
          ports:
            - name: web
              containerPort: 8080
          livenessProbe:
            tcpSocket:
              port: 80
          resources:
            requests:
              memory: "800Gi"
              cpu: "10m"
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    app: web
spec:
  selector:
    run: nginx
  ports:
    - name: http
      port: 80
      targetPort: http
EOF

Try to reach the service. It doesn't work:

oc port-forward service/my-service 8080:80

Commands to debug with

Status and events
oc get pods
oc describe pod -l app=nginx
oc get events --sort-by=.lastTimestamp
oc status --suggest
Inspect the deployment and service
oc get deployment my-deployment -o yaml
oc describe service my-service
oc get endpointslices -l kubernetes.io/service-name=my-service
oc get pods --show-labels
Look up what a field means
oc explain pod.spec.containers.resources.requests
oc explain pod.spec.containers.livenessProbe
Logs, including the previous crashed container
oc logs deployment/my-deployment
oc logs <pod-name> --previous
Start a debug copy of a pod with a shell
oc debug deployment/my-deployment
Fix things
oc edit deployment my-deployment
oc edit service my-service
Status and events
kubectl get pods
kubectl describe pod -l app=nginx
kubectl get events --sort-by=.lastTimestamp
Inspect the deployment and service
kubectl get deployment my-deployment -o yaml
kubectl describe service my-service
kubectl get endpointslices -l kubernetes.io/service-name=my-service
kubectl get pods --show-labels
Look up what a field means
kubectl explain pod.spec.containers.resources.requests
kubectl explain pod.spec.containers.livenessProbe
Logs, including the previous crashed container
kubectl logs deployment/my-deployment
kubectl logs <pod-name> --previous
Attach an ephemeral debug container to a running pod
kubectl debug -it <pod-name> --image=busybox --target=app
Fix things
kubectl edit deployment my-deployment
kubectl edit service my-service
Show the answers

There are five problems. You'll find them roughly in this order, because each one hides the next:

  1. The pod requests 800Gi of memory. No node is that big, so the pod stays Pending with FailedScheduling ... Insufficient memory. On clusters with a ResourceQuota, such as the Developer Sandbox, the pod isn't even created: oc get events shows exceeded quota for the ReplicaSet. Request something like 64Mi.
  2. The image tag 1.290 doesn't exist. Once the pod is scheduled, it shows ErrImagePull / ImagePullBackOff. Use 1.29.
  3. The liveness probe checks port 80, but nginx listens on 8080. The container is restarted over and over. Probe port 8080.
  4. The service selector is run: nginx, but the pods are labelled app: nginx, so the service has no endpoints. Select app: nginx.
  5. The service targetPort is http, but the container port is named web. Use targetPort: web or 8080.

After fixing them, oc get endpointslices -l kubernetes.io/service-name=my-service lists the pod IP, and curl localhost:8080 through the port-forward returns the nginx welcome page.

Activities

Lab Description
Lab 3 - Debugging Find and fix everything that's broken in a deployment