Health and Monitoring¶

Liveness and Readiness Probes¶

A probe is a diagnostic performed periodically by the kubelet on a container. Each probe uses one of four mechanisms:

exec: Runs a command inside the container. The check succeeds if the command exits with status code 0.

tcpSocket: Opens a TCP connection to the container's IP address on a specified port. The check succeeds if the port is open.

httpGet: Sends an HTTP GET request to the container's IP address on a specified port and path. The check succeeds if the response status code is at least 200 and below 400.

grpc: Calls the standard gRPC health checking protocol on a specified port. The check succeeds if the service reports SERVING.

The kubelet can run three kinds of probes on each container:

startupProbe: Indicates whether the application inside the container has started. Liveness and readiness probes don't run until the startup probe succeeds. If it keeps failing, the container is restarted. Use it for slow-starting applications instead of a long initialDelaySeconds.

livenessProbe: Indicates whether the container is still working. Runs for the lifetime of the container. If it fails failureThreshold times in a row, the kubelet kills the container, and it's restarted according to the pod's restartPolicy.

readinessProbe: Indicates whether the container is ready to serve requests. It also runs for the lifetime of the container. While it fails, the pod is removed from the endpoints of every Service that selects it. The container is not restarted.

The timing of every probe is tuned with initialDelaySeconds, periodSeconds (default 10), timeoutSeconds (default 1), failureThreshold (default 3) and successThreshold (default 1).

Resources¶

References¶

apiVersion: v1
kind: Pod
metadata:
  name: exec-probe
spec:
  containers:
    - name: app
      image: busybox
      command: ["sh", "-c", "echo Hello, Kubernetes! && sleep 3600"]
      livenessProbe:
        exec:
          command: ["echo", "alive"]

apiVersion: v1
kind: Pod
metadata:
  name: web-probes
spec:
  containers:
    - name: app
      image: quay.io/nginx/nginx-unprivileged:1.29
      ports:
        - containerPort: 8080
      livenessProbe:
        tcpSocket:
          port: 8080
        initialDelaySeconds: 10
      readinessProbe:
        httpGet:
          path: /
          port: 8080
        periodSeconds: 10

apiVersion: v1
kind: Pod
metadata:
  name: slow-start
spec:
  containers:
    - name: app
      image: quay.io/nginx/nginx-unprivileged:1.29
      ports:
        - containerPort: 8080
      startupProbe:          # allow up to 30 x 10s = 5 minutes to start
        httpGet:
          path: /
          port: 8080
        failureThreshold: 30
        periodSeconds: 10
      livenessProbe:         # only starts once the startup probe has passed
        httpGet:
          path: /
          port: 8080

Container Logging¶

Application and systems logs can help you understand what is happening inside your cluster. The logs are particularly useful for debugging problems and monitoring cluster activity.

Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster.

Resources¶

References¶

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
    - name: count
      image: busybox
      command:
        [
          "sh",
          "-c",
          'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 5; done',
        ]

oc logs counter                  # current logs
oc logs -f counter               # follow new lines
oc logs counter --previous       # logs from the previous (crashed) container
oc logs counter -c count         # a specific container in a multi-container pod
oc logs deployment/my-deployment # logs from a pod of a deployment

brew install stern
stern . -n default

kubectl logs counter
kubectl logs -f counter
kubectl logs counter --previous
kubectl logs counter -c count
kubectl logs deployment/my-deployment

brew install stern
stern . -n default

Monitoring Applications¶

To scale an application and provide a reliable service, you need to understand how the application behaves when it is deployed. You can examine application performance in a Kubernetes cluster by examining the containers, pods, services, and the characteristics of the overall cluster. Kubernetes provides detailed information about an application’s resource usage at each of these levels. This information allows you to evaluate your application’s performance and where bottlenecks can be removed to improve overall performance.

Prometheus, a CNCF project, can natively monitor Kubernetes, nodes, and Prometheus itself. OpenShift includes a preconfigured Prometheus-based monitoring stack, and can also monitor your own applications' metrics (user workload monitoring).

Resources¶

References¶

The following pods use the Kubernetes resource-consumer test image to burn a fixed amount of CPU, so you can watch resource usage:

apiVersion: v1
kind: Pod
metadata:
  name: cpu-500m
spec:
  containers:
    - name: app
      image: registry.k8s.io/e2e-test-images/resource-consumer:1.13
      resources:
        requests:
          cpu: 700m
          memory: 128Mi
    - name: load-generator
      image: registry.access.redhat.com/ubi9/ubi-minimal
      command:
        - /bin/sh
        - -c
        - until curl -s localhost:8080/ConsumeCPU -d "millicores=500&durationSec=3600"; do sleep 5; done && sleep infinity

apiVersion: v1
kind: Pod
metadata:
  name: cpu-200m
spec:
  containers:
    - name: app
      image: registry.k8s.io/e2e-test-images/resource-consumer:1.13
      resources:
        requests:
          cpu: 300m
          memory: 64Mi
    - name: load-generator
      image: registry.access.redhat.com/ubi9/ubi-minimal
      command:
        - /bin/sh
        - -c
        - until curl -s localhost:8080/ConsumeCPU -d "millicores=200&durationSec=3600"; do sleep 5; done && sleep infinity

oc adm top pods
oc adm top pods --containers
oc adm top nodes

oc get projects
oc api-resources -o wide
oc get nodes,ns,po,deploy,svc
oc describe node <node-name>

kubectl get --raw /apis/metrics.k8s.io/

kubectl top pods
kubectl top pods --containers
kubectl top nodes

kubectl describe node <node-name>

Activities¶