Overview

In this lab, you will configure and test Kubernetes liveness and readiness probes for containers.

Objectives

In this lab, you learn how to perform the following tasks:

• Configure and test a liveness probe
• Configure and test a readiness probe

Lab setup

Access Qwiklabs

For each lab, you get a new Google Cloud project and set of resources for a fixed time at no cost.

1. Sign in to Qwiklabs using an incognito window.

2. Note the lab's access time (for example, 1:15:00), and make sure you can finish within that time.
   There is no pause feature. You can restart if needed, but you have to start at the beginning.

3. When ready, click Start lab.

4. Note your lab credentials (Username and Password). You will use them to sign in to the Google Cloud Console.

5. Click Open Google Console.

6. Click Use another account and copy/paste credentials for this lab into the prompts.
   If you use other credentials, you'll receive errors or incur charges.

7. Accept the terms and skip the recovery resource page.

After you complete the initial sign-in steps, the project dashboard appears.

Activate Google Cloud Shell

Google Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud.

Google Cloud Shell provides command-line access to your Google Cloud resources.

1. In Cloud console, on the top right toolbar, click the Open Cloud Shell button.

   

2. Click Continue.

It takes a few moments to provision and connect to the environment. When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. For example:

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

• You can list the active account name with this command:

gcloud auth list

Output:

Credentialed accounts: - @.com (active)

Example output:

Credentialed accounts: - google1623327_student@qwiklabs.net

• You can list the project ID with this command:

gcloud config list project

Output:

[core] project =

Example output:

[core] project = qwiklabs-gcp-44776a13dea667a6 Note: Full documentation of gcloud is available in the gcloud CLI overview guide .

Task 1. Connect to the lab GKE Cluster

In this task, you connect to the lab GKE Cluster and clone the lab repository to the lab cloud shell.

Connect to the lab GKE Cluster

1. In Cloud Shell, type the following command to set the environment variable for the zone and cluster name:

export my_zone={{{ project_0.default_zone | "ZONE" }}} export my_cluster=standard-cluster-1

2. Configure tab completion for the kubectl command-line tool:

source <(kubectl completion bash)

3. Configure access to your cluster for kubectl:

gcloud container clusters get-credentials $my_cluster --zone $my_zone

4. In Cloud Shell enter the following command to clone the lab repository to the lab Cloud Shell:

git clone https://github.com/GoogleCloudPlatform/training-data-analyst

5. Create a soft link as a shortcut to the working directory:

ln -s ~/training-data-analyst/courses/ak8s/v1.1 ~/ak8s

6. Change to the directory that contains the sample files for this lab:

cd ~/ak8s/Probes/

Task 2. Configure liveness probes

Configure a liveness probe

In this task, you will deploy a liveness probe to detect applications that have transitioned from a running state to a broken state. Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup.

In such cases, you don't want to kill the application, but you don't want to send it requests either. Kubernetes provides readiness probes to detect and mitigate these situations. A Pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.

Readiness probes are configured similarly to liveness probes. The only difference is that you use the readinessProbe field instead of the livenessProbe field.

A Pod definition file called exec-liveness.yaml has been provided for you that defines a simple container called liveness running Busybox and a liveness probe that uses the cat command against the file /tmp/healthy within the container to test for liveness every 5 seconds.

The startup script for the liveness container creates the /tmp/healthy on startup and then deletes it 30 seconds later to simulate an outage that the Liveness probe can detect.

apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: - name: liveness image: k8s.gcr.io/busybox args: - /bin/sh - -c - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5

1. In the Cloud Shell, execute the following command to create the Pod:

kubectl create -f exec-liveness.yaml

2. Within 30 seconds, view the Pod events:

kubectl describe pod liveness-exec

The output indicates that no liveness probes have failed yet.

Condensed sample output:

Type: Secret (a volume populated by a Secret) SecretName: default-token-wq52t Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age ... Message ---- ------ ---- ... ------- Normal Scheduled 11s ... Successfully assigned liveness-e ... Normal Su...ntVolume 10s ... MountVolume.SetUp succeeded for ... Normal Pulling 10s ... pulling image "k8s.gcr.io/busybox" Normal Pulled 9s ... Successfully pulled image "k8s.g ... Normal Created 9s ... Created container Normal Started 9s ... Started container

The sample output shown here has been condensed for readability, you will see a lot more detail on screen.

3. After 35 seconds, view the Pod events again:

kubectl describe pod liveness-exec

At the bottom of the output, there are messages indicating that the liveness probes have failed, and the containers have been killed and recreated.

Condensed sample output:

Type: Secret (a volume populated by a Secret) SecretName: default-token-wq52t Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age ... Message ---- ------ ---- ... ------- Normal Scheduled 2m ... Successfully assigned liveness-e ... Normal Su...ntVolume 2m ... MountVolume.SetUp succeeded for ... Normal Pulling 49s ... pulling image "k8s.gcr.io/busybox" Normal Pulled 49s ... Successfully pulled image "k8s.g ... Normal Created 49s ... Created container Normal Started 49s ... Started container Normal Killing 49s ... Killing container with id docker://liveness:Container failed liveness probe.. Container will be killed and recreated. Warning Unhealthy 5s ... Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory

4. Wait another 60 seconds, and verify that the Container has been restarted:

kubectl get pod liveness-exec

The output shows that RESTARTS has been incremented in response to the failure detected by the liveness probe:

Sample output:

NAME READY STATUS RESTARTS AGE liveness-exec 1/1 Running 2 4m

5. Delete the liveness probe demo pod:

kubectl delete pod liveness-exec

Click Check my progress to verify the objective. Configure Liveness Probes

Task 3. Configure readiness probes

Although a pod could successfully start and be considered healthy by a liveness probe, it's likely that it may not be ready to receive traffic right away. This is common for deployments that serve as a backend to a service such as a load balancer. A readiness probe is used to determine when a pod and its containers are ready to begin receiving traffic.

Readiness probes are configured similarly to liveness probes. The only difference in configuration is that you use the readinessProbe field instead of the livenessProbe field. Readiness probes control whether a specific container is considered ready, and this is used by services to decide when a container can have traffic sent to it.

In this task a Pod definition file called readiness-deployment.yaml has been provided for you that create a single pods that will serve as a test web server along with a load balancer

The container has a readiness probe defined that uses the cat command against the file /tmp/healthz within the container to test for readiness every 5 seconds.

Each container also has a liveness probe defined that uses the cat command against the same file within the container to test for readiness every 5 seconds but it also has a startup delay of 45 seconds to simulate an application that might have a complex startup process that takes time to stabilize after a container has started.

Once the service has started handling traffic this pattern ensures that the service will forward traffic only to containers that are ready to handle traffic.

apiVersion: v1 kind: Pod metadata: labels: demo: readiness-probe name: readiness-demo-pod spec: containers: - name: readiness-demo-pod image: nginx ports: - containerPort: 80 readinessProbe: exec: command: - cat - /tmp/healthz initialDelaySeconds: 5 periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: name: readiness-demo-svc labels: demo: readiness-probe spec: type: LoadBalancer ports: - port: 80 targetPort: 80 protocol: TCP selector: demo: readiness-probe

1. In the Cloud Shell, execute the following command to create the Pod:

kubectl create -f readiness-deployment.yaml

2. In the Cloud Shell, check the status of the readiness-demo-svc load balancer service:

kubectl get service readiness-demo-svc

3. Enter the IP address in a browser window and you'll notice that you'll get an error message signifying that the site cannot be reached.

4. Check the pod's events:

kubectl describe pod readiness-demo-pod

The output will reveal that the readiness probe has failed:

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m24s default-scheduler Successfully assigned default/readiness-demo-pod to gke-load-test-default-pool-abd43157-rgg0 Normal Pulling 2m23s kubelet Pulling image "nginx" Normal Pulled 2m23s kubelet Successfully pulled image "nginx" Normal Created 2m23s kubelet Created container readiness-demo-pod Normal Started 2m23s kubelet Started container readiness-demo-pod Warning Unhealthy 35s (x21 over 2m15s) kubelet Readiness probe failed: cat: /tmp/healthz: No such file or directory

Unlike the liveness probe, an unhealthy readiness probe does not trigger the pod to restart.

5. Use the following command to generate the file that the readiness probe is checking for:

kubectl exec readiness-demo-pod -- touch /tmp/healthz

The Conditions section of the pod description should now show True as the value for Ready.

kubectl describe pod readiness-demo-pod | grep ^Conditions -A 5

Output:

Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True

6. Now, refresh the browser tab that had your readiness-demo-svc external IP. You should see a "Welcome to nginx!" message properly displayed.

Setting meaningful readiness probes for your application containers ensures that pods are only receiving traffic when they are ready to do so. An example of a meaningful readiness probe is checking to see whether a cache your application relies on is loaded at startup.

Click Check my progress to verify the objective. Configure Readiness Probes

Monitor the behavior of the liveness and readiness probes

You can now check how the ready status of the Pods in the deployment corresponds to the endpoints that are actively enabled for your service. As Pods fail the readiness and liveness probe tests they are marked as not ready, their endpoints are removed from the service, and the liveness probe initiates the restart process to recover the failed pod.

The restarted pods are not marked as ready immediately and have to wait for the readiness test to pass before the service will add the endpoint back into it's pool.

1. Now check the status of the pods again:

kubectl get pods

You will now see that the individual pods are shown as ready. Depending on timing one or more of these might restart but it should take 2-3 minutes for that to happen. Pods that are listed as ready should have a corresponding endpoint associated with the service.

2. To check that you can connect to the application first query the external ip-address from the load balancer service details and save it in an environment variable:

export EXTERNAL_IP=$(kubectl get services readiness-demo-svc -o json | jq -r '.status.loadBalancer.ingress[0].ip')

3. Check to see if the application is responding:

curl $EXTERNAL_IP

4. Over the course of the next few minutes repeat the following commands every 10 seconds or so to check the deployment status and send a request to the load balancer:

curl $EXTERNAL_IP kubectl get pods

You should continue to see responses with no failures or timeouts even though individual containers are being restarted regularly due to the failures detected by the liveness probes. If all three containers restart at more or less the same time you might still see a timeout but that should rarely happen.

The combination of the liveness and readiness probes provides a way to ensure that failed systems are restarted safely while the service only forwards traffic on to containers that are known to be able to respond.

End your lab

When you have completed your lab, click End Lab. Google Cloud Skills Boost removes the resources you’ve used and cleans the account for you.

You will be given an opportunity to rate the lab experience. Select the applicable number of stars, type a comment, and then click Submit.

The number of stars indicates the following:

• 1 star = Very dissatisfied
• 2 stars = Dissatisfied
• 3 stars = Neutral
• 4 stars = Satisfied
• 5 stars = Very satisfied

You can close the dialog box if you don't want to provide feedback.

For feedback, suggestions, or corrections, please use the Support tab.

Copyright 2022 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.