= Kubernetes Concepts :toc: :icons: :linkattrs: :imagesdir: ../../resources/images == Introduction Now that we have a cluster up and running we can start exploring the Kubernetes CLI via the `kubectl` (pronounced "cube control") command. `kubectl` interacts with the Kubernetes API Server, which runs on the master nodes in the cluster. Kubernetes as a platform has a number of abstractions that map to API objects. These Kubernetes API Objects can be used to describe your cluster's desired state - including information such as applications and workloads running, container images, networking resources, and more. This section explains the most-used Kubernetes API concepts and how to interact with them via `kubectl`. == Prerequisites This chapter uses an EKS cluster with worker nodes as described link:../102-your-first-cluster[here]. [NOTE] This lab and future labs can be completed using the EKS cluster or the multi-master kops cluster. Some of the outputs may vary slightly if using the kops cluster. All configuration files for this chapter are in the `01-path-basics/103-kubernetes-concepts/templates` directory. Please be sure to `cd` into that directory before running the commands below. $ cd ~/environment/aws-workshop-for-kubernetes/01-path-basics/103-kubernetes-concepts/templates == Display Nodes This command will show all the nodes available in your kubernetes cluster: $ kubectl get nodes It will show an output similar to: NAME STATUS ROLES AGE VERSION ip-192-168-160-85.us-west-2.compute.internal Ready 10m v1.10.3 ip-192-168-229-150.us-west-2.compute.internal Ready 10m v1.10.3 ip-192-168-79-105.us-west-2.compute.internal Ready 10m v1.10.3 If you do not see this output, or receive an error, please ensure that you've followed the steps link:../102-your-first-cluster[here] and have a validated cluster. === Create your first Pod This command instantiates an nginx container into your cluster, inside a pod: $ kubectl run nginx --image=nginx deployment "nginx" created Get the list of deployments: $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx 1 1 1 0 41s Get the list of running pods: $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-4217019353-pmkzb 1/1 Running 0 1m Get additional details for the pod by using the `` from the above output: ``` $ kubectl describe pod nginx-65899c769f-nd8qk Name: nginx-65899c769f-nd8qk Namespace: default Node: ip-192-168-229-150.us-west-2.compute.internal/192.168.229.150 Start Time: Wed, 06 Jun 2018 11:55:08 +0000 Labels: pod-template-hash=2145573259 run=nginx Annotations: Status: Running IP: 192.168.217.248 Controlled By: ReplicaSet/nginx-65899c769f Containers: nginx: Container ID: docker://fae3055b8829a6395d60a4f78cc38e6d4d7439dfb9b13e3e2c7bea2c550bee68 Image: nginx Image ID: docker-pullable://nginx@sha256:6be552b9a3c4762ae34962ddcb0e1aed3dfd0513860a846777182030b1a1bd0c Port: Host Port: State: Running Started: Wed, 06 Jun 2018 11:55:14 +0000 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-prvj5 (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: default-token-prvj5: Type: Secret (a volume populated by a Secret) SecretName: default-token-prvj5 Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 1m default-scheduler Successfully assigned nginx-65899c769f-nd8qk to ip-192-168-229-150.us-west-2.compute.internal Normal SuccessfulMountVolume 1m kubelet, ip-192-168-229-150.us-west-2.compute.internal MountVolume.SetUp succeeded for volume "default-token-prvj5" Normal Pulling 1m kubelet, ip-192-168-229-150.us-west-2.compute.internal pulling image "nginx" Normal Pulled 1m kubelet, ip-192-168-229-150.us-west-2.compute.internal Successfully pulled image "nginx" Normal Created 1m kubelet, ip-192-168-229-150.us-west-2.compute.internal Created container Normal Started 1m kubelet, ip-192-168-229-150.us-west-2.compute.internal Started container ``` By default, pods are created in a `default` namespace. In addition, a `kube-system` namespace is also reserved for Kubernetes system pods. A list of all the pods in `kube-system` namespace can be displayed as shown: ``` $ kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE aws-node-76w4v 1/1 Running 0 16m aws-node-m55x9 1/1 Running 1 16m aws-node-wxd2z 1/1 Running 0 16m kube-dns-7cc87d595-d95l5 3/3 Running 0 1h kube-proxy-dq4jx 1/1 Running 0 16m kube-proxy-kq5f9 1/1 Running 0 16m kube-proxy-rgxn7 1/1 Running 0 16m ``` Again, the exact output may vary but your results should look similar to these. === Get logs from the pod Logs from the pod can be obtained (a fresh nginx does not have logs - check again later once you have accessed the service): $ kubectl logs --namespace === Execute a shell on the running pod This command will open a TTY to a shell in your pod: $ kubectl get pods $ kubectl exec -it /bin/bash This opens a bash shell and allows you to look around the filesystem of the container. === Clean up Delete all the Kubernetes resources created so far: $ kubectl delete deployment/nginx In the next sections, we will go into more detail about Pods, Deployments, and other commonly used Kubernetes objects. == Pods A Pod is the smallest deployable unit that can be created, scheduled, and managed. It’s a logical collection of containers that belong to an application. Pods are created in a namespace. All containers in a pod share the namespace, volumes and networking stack. This allows containers in the pod to "`find`" each other and communicate using `localhost`. === Create a Pod Each resource in Kubernetes can be defined using a configuration file. For example, an NGINX pod can be defined with configuration file shown in below: $ cat pod.yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod labels: name: nginx-pod spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 Create the pod as shown below: $ kubectl apply -f pod.yaml pod "nginx-pod" created Get the list of pod: $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-pod 1/1 Running 0 22s Verify that the pod came up fine (ensure nothing else is running on port 8080): $ kubectl -n default port-forward $(kubectl -n default get pod -l name=nginx-pod -o jsonpath='{.items[0].metadata.name}') 8080:80 In your Cloud9 IDE, click **Preview** and **Preview Running Application**. This opens up a preview tab and shows the NGINX main page: image::nginx-pod-default-page.png[] If the containers in the pod generate logs, then they can be seen using the command shown: $ kubectl logs nginx-pod 127.0.0.1 - - [03/Nov/2017:17:33:30 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" "-" 127.0.0.1 - - [03/Nov/2017:17:33:32 +0000] "GET /favicon.ico HTTP/1.1" 404 571 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" "-" 2017/11/03 17:33:32 [error] 5#5: *2 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8080", referrer: "http://localhost:8080/" === Memory and CPU resource request A Container in a Pod can be assigned memory and CPU _request_ and _limit_. Request is the minimum amount of memory/CPU that Kubernetes will give to the container. Limit is the maximum amount of memory/CPU that a container will be allowed to use. The memory/CPU request/limit for the Pod is the sum of the memory/CPU requests/limits for all the Containers in the Pod. Request defaults to limit if not specified. Default value of the limit is the node capacity. A Pod can be scheduled on a node if the Pod's memory and CPU request can be met. Memory and CPU limits are not taken into consideration for scheduling. Pod can continue to operate on the node if Containers in the Pod does not exceed the memory request. If Containers in the Pod exceeds the memory request then they become target of eviction whenever the node runs out of memory. If Containers in the Pod exceeds the memory limit then they are terminated. If the Pod can be restarted, then kubelet will restart it, just like any other type of runtime failure. A Container might or might not be allowed to exceed its CPU limit for extended periods of time. However, it will not be killed for excessive usage. Memory and CPU request/limit can be specified using the following: [options="header", width="75%", cols="1,3"] |==== | Type | Field | Memory request | `spec.containers[].resources.requests.memory` | Memory limit | `spec.containers[].resources.limits.memory` | CPU request | `spec.containers[].resources.requests.cpu` | CPU limit | `spec.containers[].resources.limits.cpu` |==== Memory resources are requested in bytes. You can specify them in integer or decimals with one of the suffixes `E`, `P`, `T`, `G`, `M`, `K`. It can also be expressed with power-of-two equivalents `Ei`, `Pi`, `Ti`, `Gi`, `Mi`, `Ki`. CPU can be requested in _cpu units_. 1 cpu unit is equivalent 1 AWS vCPU. It can also be requested in fractional units, such as 0.5 or in _millicpu_ such as 500m. ===== Default memory and CPU By default, a container in a pod is not allocated any requests or limits. This can be verified using the previously started pod: $ kubectl get pod/nginx-pod -o jsonpath={.spec.containers[].resources} map[] ===== Assign memory and CPU Let's assign a memory request and limit to a Pod using the configuration file shown: $ cat pod-resources.yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod2 labels: name: nginx-pod spec: containers: - name: nginx image: nginx:latest resources: limits: memory: "200Mi" cpu: 2 requests: memory: "100Mi" cpu: 1 ports: - containerPort: 80 The only change in this configuration file is the addition of `spec.containers[].resources` section. The limits are specified in the `limits` section and the requests are specified in the `requests` section. Create the pod: $ kubectl apply -f pod-resources.yaml pod "nginx-pod2" created Get more details about the requests and limits: $ kubectl get pod/nginx-pod2 -o jsonpath={.spec.containers[].resources} map[limits:map[memory:200Mi cpu:2] requests:map[cpu:1 memory:100Mi]] NGINX container requires fairly low memory and CPU. And so these request and limit numbers would work well, and the pod is started correctly. Now, let's try to start a WildFly pod using similar numbers. The configuration file for the same is shown: $ cat pod-resources1.yaml apiVersion: v1 kind: Pod metadata: name: wildfly-pod labels: name: wildfly-pod spec: containers: - name: wildfly image: jboss/wildfly:11.0.0.Final resources: limits: memory: "200Mi" cpu: 2 requests: memory: "100Mi" cpu: 1 ports: - containerPort: 8080 The max amount of memory allocated for the WildFly container in this pod is restricted to 200MB. Let's create this Pod: $ kubectl apply -f pod-resources1.yaml pod "wildfly-pod" created Watch the status of the Pod: $ kubectl get pods -w NAME READY STATUS RESTARTS AGE wildfly-pod 0/1 ContainerCreating 0 5s wildfly-pod 1/1 Running 0 26s wildfly-pod 0/1 OOMKilled 0 29s wildfly-pod 1/1 Running 1 31s wildfly-pod 0/1 OOMKilled 1 34s wildfly-pod 0/1 CrashLoopBackOff 1 45s wildfly-pod 1/1 Running 2 46s wildfly-pod 0/1 OOMKilled 2 49s wildfly-pod 0/1 CrashLoopBackOff 2 1m wildfly-pod 1/1 Running 3 1m wildfly-pod 0/1 OOMKilled 3 1m `OOMKilled` shows that the container was terminated because it ran out of memory. To correct this, we'll need to re-create the pod with higher memory limits. Although it may be instinctive to simply adjust the memory limit in the existing pod definition and re-apply it, Kubernetes does not currently support changing resource limits on running pods, so we'll need to first delete the existing pod, then recreate it. In `pod-resources2.yaml`, confirm that the value of `spec.containers[].resources.limits.memory` is `300Mi`. Delete the existing Pod, and create a new one: $ kubectl delete -f pod-resources1.yaml pod "wildfly-pod" deleted $ kubectl apply -f pod-resources2.yaml pod "wildfly-pod" created $ kubectl get -w pod/wildfly-pod NAME READY STATUS RESTARTS AGE wildfly-pod 0/1 ContainerCreating 0 3s wildfly-pod 1/1 Running 0 25s Now, the Pod successfully starts. Get more details about the resources allocated to the Pod: $ kubectl get pod/wildfly-pod -o jsonpath={.spec.containers[].resources} map[limits:map[cpu:2 memory:300Mi] requests:map[cpu:1 memory:100Mi]] === Quality of service Kubernetes opportunistically scavenges the difference between request and limit if they are not used by the Containers. This allows Kubernetes to oversubscribe nodes, which increases utilization, while at the same time maintaining resource guarantees for the containers that need guarantees. Kubernetes assigns one of the QoS classes to the Pod: . `Guaranteed` . `Burstable` . `BestEffort` QoS class is used by Kubernetes for scheduling and evicting Pods. When every Container in a Pod is given a memory and CPU limit, and optionally non-zero request, and they exactly match, then a Pod is scheduled with `Guaranteed` QoS. This is the highest priority. A Pod is given `Burstable` QoS class if the Pod does not meet the `Guaranteed` QoS and at least one Container has a memory or CPU request. This is intermediate priority. When no memory and CPU request or limit is assigned to any Container in the Pod, then a Pod is scheduled with `BestEffort` QoS. This the lowest and the default priority. Pods that need to stay up can request `Guaranteed` QoS. Pods with less stringent requirement can use a weaker or no QoS. ==== Guaranteed Here is an example of Pod with `Guaranteed` QoS: $ cat pod-guaranteed.yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod-guaranteed labels: name: nginx-pod spec: containers: - name: nginx image: nginx:latest resources: limits: memory: "200Mi" cpu: 1 ports: - containerPort: 80 Note that no request values are specified here, and will default to limit. Create this Pod: $ kubectl apply -f pod-guaranteed.yaml pod "nginx-pod-guaranteed" created Check the resources: $ kubectl get pod/nginx-pod-guaranteed -o jsonpath={.spec.containers[].resources} map[limits:map[cpu:1 memory:200Mi] requests:map[cpu:1 memory:200Mi]] Check the QoS: $ kubectl get pod/nginx-pod-guaranteed -o jsonpath={.status.qosClass} Guaranteed Another Pod with explicit value for limit and request is shown: $ cat pod-guaranteed2.yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod-guaranteed2 labels: name: nginx-pod spec: containers: - name: nginx image: nginx:latest resources: limits: memory: "200Mi" cpu: 1 requests: memory: "200Mi" cpu: 1 ports: - containerPort: 80 Create this Pod: $ kubectl apply -f pod-guaranteed2.yaml pod "nginx-pod-guaranteed2" created Check the resources: $ kubectl get pod/nginx-pod-guaranteed2 -o jsonpath={.spec.containers[].resources} map[limits:map[cpu:1 memory:200Mi] requests:map[cpu:1 memory:200Mi]] Check the QoS: $ kubectl get pod/nginx-pod-guaranteed2 -o jsonpath={.status.qosClass} Guaranteed ==== Burstable Here is an example of Pod with `Burstable` QoS: $ cat pod-burstable.yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod-burstable labels: name: nginx-pod spec: containers: - name: nginx image: nginx:latest resources: limits: memory: "200Mi" cpu: 1 requests: memory: "100Mi" cpu: 1 ports: - containerPort: 80 Note that both request and limit values are specified here. Create this Pod: $ kubectl apply -f pod-burstable.yaml pod "nginx-pod-burstable" created Check the resources: $ kubectl get pod/nginx-pod-burstable -o jsonpath={.spec.containers[].resources} map[limits:map[cpu:1 memory:200Mi] requests:map[cpu:1 memory:100Mi]] Check the QoS: $ kubectl get pod/nginx-pod-burstable -o jsonpath={.status.qosClass} Burstable ==== BestEffort Check the resources: $ kubectl get pod/nginx-pod -o jsonpath={.spec.containers[].resources} map[requests:map[cpu:100m]] Check the QoS: $ kubectl get pod/nginx-pod -o jsonpath={.status.qosClass} Burstable This should be `BestEffort` and filed as https://github.com/kubernetes/kubernetes/issues/55278[kubernetes#55278]. === Delete a Pod Get all the Pods that are running: $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-pod 1/1 Running 0 6m nginx-pod-burstable 1/1 Running 0 9m nginx-pod-guaranteed 1/1 Running 0 23m nginx-pod-guaranteed2 1/1 Running 0 12m nginx-pod2 1/1 Running 0 6m wildfly-pod 1/1 Running 0 6m Delete the Pods as shown below: $ kubectl delete $(kubectl get pods -o=name) pod "nginx-pod" deleted pod "nginx-pod-burstable" deleted pod "nginx-pod-guaranteed" deleted pod "nginx-pod-guaranteed2" deleted pod "nginx-pod2" deleted pod "wildfly-pod" deleted == Deployment A "`desired state`", such as 4 replicas of a pod, can be described in a Deployment object. The Deployment controller in Kubernetes cluster then ensures the desired and the actual state are matching. Deployment ensures the recreation of a pod when the worker node fails or reboots. If a pod dies, then a new pod is started to ensure the desired vs actual matches. It also allows both up- and down-scaling the number of replicas. This is achieved using ReplicaSet. The Deployment manages the ReplicaSets and provides updates to those pods. === Create a Deployment The folowing example will create a Deployment with 3 replicas of NGINX base image. Let's begin with the template: $ cat deployment.yaml apiVersion: apps/v1 kind: Deployment # kubernetes object type metadata: name: nginx-deployment # deployment name spec: replicas: 3 # number of replicas selector: matchLabels: app: nginx template: metadata: labels: app: nginx # pod labels spec: containers: - name: nginx # container name image: nginx:1.12.1 # nginx image imagePullPolicy: IfNotPresent # if exists, will not pull new image ports: # container and host port assignments - containerPort: 80 - containerPort: 443 This deployment will create 3 instances of NGINX image. Run the following command to create Deployment: $ kubectl create -f deployment.yaml --record deployment "nginx-deployment" created The `--record` flag will track changes made through each revision. To monitor deployment rollout status: $ kubectl rollout status deployment/nginx-deployment deployment "nginx-deployment" successfully rolled out A Deployment creates a ReplicaSet to manage the number of replicas. Let's take a look at existing deployments and replica set. Get the deployments: $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 3 3 3 3 25s Get the replica set for the deployment: $ kubectl get replicaset NAME DESIRED CURRENT READY AGE nginx-deployment-3441592026 3 3 3 1m Get the list of running pods: $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-3441592026-ddpf0 1/1 Running 0 2m nginx-deployment-3441592026-kkp8h 1/1 Running 0 2m nginx-deployment-3441592026-lx304 1/1 Running 0 2m === Scaling a Deployment Number of replicas for a Deployment can be scaled using the following command: $ kubectl scale --replicas=5 deployment/nginx-deployment deployment "nginx-deployment" scaled Verify the deployment: $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 5 5 5 5 2m Verify the pods in the deployment: $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-3441592026-36957 1/1 Running 0 44s nginx-deployment-3441592026-8wch5 1/1 Running 0 44s nginx-deployment-3441592026-ddpf0 1/1 Running 0 3m nginx-deployment-3441592026-kkp8h 1/1 Running 0 3m nginx-deployment-3441592026-lx304 1/1 Running 0 3m === Update a Deployment A more general update to Deployment can be made by making edits to the pod spec. In this example, let's change to the latest nginx image. First, type the following to open up a text editor: $ kubectl edit deployment/nginx-deployment Next, change the image from `nginx:1.12.1` to `nginx:latest`. This should perform a rolling update of the deployment. To track the deployment details such as revision, image version, and ports - type in the following: ``` $ kubectl describe deployments Name: nginx-deployment Namespace: default CreationTimestamp: Mon, 23 Oct 2017 09:14:36 -0400 Labels: app=nginx Annotations: deployment.kubernetes.io/revision=2 kubernetes.io/change-cause=kubectl edit deployment/nginx-deployment Selector: app=nginx Replicas: 5 desired | 5 updated | 5 total | 5 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 1 max unavailable, 1 max surge Pod Template: Labels: app=nginx Containers: nginx: Image: nginx:latest Ports: 80/TCP, 443/TCP Environment: Mounts: Volumes: Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable OldReplicaSets: NewReplicaSet: nginx-deployment-886641336 (5/5 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 4m deployment-controller Scaled up replica set nginx-deployment-3441592026 to 3 Normal ScalingReplicaSet 1m deployment-controller Scaled up replica set nginx-deployment-3441592026 to 5 Normal ScalingReplicaSet 32s deployment-controller Scaled up replica set nginx-deployment-886641336 to 1 Normal ScalingReplicaSet 32s deployment-controller Scaled down replica set nginx-deployment-3441592026 to 4 Normal ScalingReplicaSet 32s deployment-controller Scaled up replica set nginx-deployment-886641336 to 2 Normal ScalingReplicaSet 29s deployment-controller Scaled down replica set nginx-deployment-3441592026 to 3 Normal ScalingReplicaSet 29s deployment-controller Scaled up replica set nginx-deployment-886641336 to 3 Normal ScalingReplicaSet 28s deployment-controller Scaled down replica set nginx-deployment-3441592026 to 2 Normal ScalingReplicaSet 28s deployment-controller Scaled up replica set nginx-deployment-886641336 to 4 Normal ScalingReplicaSet 25s (x3 over 26s) deployment-controller (combined from similar events): Scaled down replica set nginx-deployment-3441592026 to 0 ``` === Rollback a Deployment To rollback to a previous version, first check the revision history: $ kubectl rollout history deployment/nginx-deployment deployments "nginx-deployment" REVISION CHANGE-CAUSE 1 kubectl scale deployment/nginx-deployment --replicas=5 2 kubectl edit deployment/nginx-deployment If you only want to rollback to the previous revision, enter the following command: $ kubectl rollout undo deployment/nginx-deployment deployment "nginx-deployment" rolled back In our case, the deployment will rollback to use the `nginx:1.12.1` image. Check the image name: $ kubectl describe deployments | grep Image Image: nginx:1.12.1 If rolling back to a specific revision then enter: $ kubectl rollout undo deployment/nginx-deployment --to-revision= === Delete a Deployment Run the following command to delete the Deployment: $ kubectl delete -f deployment.yaml deployment "nginx-deployment" deleted == Service A pod is ephemeral. Each pod is assigned a unique IP address. If a pod that belongs to a replication controller dies, then it is recreated and may be given a different IP address. Further, additional pods may be created using Deployment or Replica Set. This makes it difficult for an application server, such as WildFly, to access a database, such as MySQL, using its IP address. A Service is an abstraction that defines a logical set of pods and a policy by which to access them. The IP address assigned to a service does not change over time, and thus can be relied upon by other pods. Typically, the pods belonging to a service are defined by a label selector. This is similar mechanism to how pods belong to a replica set. This abstraction of selecting pods using labels enables a loose coupling. The number of pods in the deployment may scale up or down but the application server can continue to access the database using the service. A Kubernetes service defines a logical set of pods and enables them to be accessed through microservices. === Create a Deployment for Service Pods belong to a service by using a loosely-coupled model where labels are attached to a pod and a service picks the pods by using those labels. Let's create a Deployment first that will create 3 replicas of a pod: $ cat echo-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: echo-deployment spec: replicas: 3 selector: matchLabels: app: echo-pod template: metadata: labels: app: echo-pod spec: containers: - name: echoheaders image: k8s.gcr.io/echoserver:1.10 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 This example creates an echo app that responds with HTTP headers from an Elastic Load Balancer. Type the following to create the deployment: $ kubectl create -f echo-deployment.yaml --record Use the `kubectl describe deployment` command to confirm `echo-app` has been deployed: ``` $ kubectl describe deployment Name: echo-deployment Namespace: default CreationTimestamp: Mon, 23 Oct 2017 10:07:47 -0400 Labels: app=echo-pod Annotations: deployment.kubernetes.io/revision=1 kubernetes.io/change-cause=kubectl create --filename=templates/echo-deployment.yaml --record=true Selector: app=echo-pod Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 1 max unavailable, 1 max surge Pod Template: Labels: app=echo-pod Containers: echoheaders: Image: k8s.gcr.io/echoserver:1.10 Port: 8080/TCP Environment: Mounts: Volumes: Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable OldReplicaSets: NewReplicaSet: echo-deployment-3396249933 (3/3 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 10s deployment-controller Scaled up replica set echo-deployment-3396249933 to 3 ``` Get the list of pods: ``` $ kubectl get pods NAME READY STATUS RESTARTS AGE echo-deployment-3396249933-8slzp 1/1 Running 0 1m echo-deployment-3396249933-bjwqj 1/1 Running 0 1m echo-deployment-3396249933-r05nr 1/1 Running 0 1m ``` Check the label for a pod: ``` $ kubectl describe pods/echo-deployment-3396249933-8slzp | grep Label Labels: app=echo-pod ``` Each pod in this deployment has `app=echo-pod` label attached to it. === Create a Service In the following example, we create a service `echo-service`: $ cat service.yaml apiVersion: v1 kind: Service metadata: name: echo-service spec: selector: app: echo-pod ports: - name: http protocol: TCP port: 80 targetPort: 8080 type: LoadBalancer The set of pods targeted by the service are determined by the label `app: echo-pod` attached to them. It also defines an inbound port 80 to the target port of 8080 on the container. Kubernetes supports both TCP and UDP protocols. === Publish a Service A service can be published to an external IP using the `type` attribute. This attribute can take one of the following values: . `ClusterIP`: Service exposed on an IP address inside the cluster. This is the default behavior. . `NodePort`: Service exposed on each Node's IP address at a defined port. . `LoadBalancer`: If deployed in the cloud, exposed externally using a cloud-specific load balancer. . `ExternalName`: Service is attached to the `externalName` field. It is mapped to a CNAME with the value. Let's publish the service load balancer and expose your services, add a `type` field of `LoadBalancer`. This template will expose `echo-app` service on an Elastic Load Balancer (ELB): $ cat service.yaml apiVersion: v1 kind: Service metadata: name: echo-service spec: selector: app: echo-pod ports: - name: http protocol: TCP port: 80 targetPort: 8080 type: LoadBalancer Run the following command to create the service: $ kubectl create -f service.yaml --record Get more details about the service: ``` $ kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE echo-service LoadBalancer 10.100.114.136 aefc2ab9c6985... 80:30730/TCP 12m kubernetes ClusterIP 10.100.0.1 443/TCP 1h $ kubectl describe service echo-service Name: echo-service Namespace: default Labels: Annotations: kubernetes.io/change-cause=kubectl create --filename=service.yaml --record=true Selector: app=echo-pod Type: LoadBalancer IP: 10.100.114.136 LoadBalancer Ingress: aefc2ab9c698511e88f6106a0b8e1215-513633583.us-west-2.elb.amazonaws.com Port: http 80/TCP TargetPort: 8080/TCP NodePort: http 30730/TCP Endpoints: 192.168.165.191:8080,192.168.193.104:8080,192.168.97.237:8080 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal EnsuringLoadBalancer 16s service-controller Ensuring load balancer Normal EnsuredLoadBalancer 14s service-controller Ensured load balancer ``` The output shows `LoadBalancer Ingress` as the address of an Elastic Load Balancer (ELB). It takes about 2-3 minutes for the ELB to be provisioned and be available. Wait for a couple of minutes, and then access the service: ``` curl aefc2ab9c698511e88f6106a0b8e1215-513633583.us-west-2.elb.amazonaws.com Hostname: echo-deployment-7bcf9557cc-bfdls Pod Information: -no pod information available- Server values: server_version=nginx: 1.13.3 - lua: 10008 Request Information: client_address=192.168.79.105 method=GET real path=/ query= request_version=1.1 request_scheme=http request_uri=http://aefc2ab9c698511e88f6106a0b8e1215-513633583.us-west-2.elb.amazonaws.com:8080/ Request Headers: accept=*/* host=aefc2ab9c698511e88f6106a0b8e1215-513633583.us-west-2.elb.amazonaws.com user-agent=curl/7.53.1 Request Body: -no body in request- ``` Note the `client_address` value shown in the output. This is the IP address of the pod serving the request. Multiple invocations of this command will show different values for this attribute. Now, the number of pods in the deployment can be scaled up and down. Or the pods may terminate and restart on a different host. But the service will still be able to target those pods because of the labels attached to the pod and used by the service. === Delete a Service Run the following command to delete the Service: $ kubectl delete -f service.yaml The backend Deployment needs to be explicitly deleted as well: $ kubectl delete -f echo-deployment.yaml == Daemon Set Daemon Set ensure that a copy of the pod runs on a selected set of nodes. By default, all nodes in the cluster are selected. A selection critieria may be specified to select a limited number of nodes. As new nodes are added to the cluster, pods are started on them. As nodes are removed, pods are removed through garbage collection. === Create a DaemonSet The following is an example DaemonSet that runs a Prometheus container. Let's begin with the template: $ cat daemonset.yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: prometheus-daemonset spec: selector: matchLabels: tier: monitoring name: prometheus-exporter template: metadata: labels: tier: monitoring name: prometheus-exporter spec: containers: - name: prometheus image: prom/node-exporter ports: - containerPort: 80 Run the following command to create the ReplicaSet and pods: $ kubectl create -f daemonset.yaml --record The `--record` flag will track changes made through each revision. Get basic details about the DaemonSet: $ kubectl get daemonsets/prometheus-daemonset NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE prometheus-daemonset 3 3 3 3 3 7s Get more details about the DaemonSet: ``` $ kubectl describe daemonset/prometheus-daemonset Name: prometheus-daemonset Selector: name=prometheus-exporter,tier=monitoring Node-Selector: Labels: name=prometheus-exporter tier=monitoring Annotations: kubernetes.io/change-cause=kubectl create --filename=templates/daemonset.yaml --record=true Desired Number of Nodes Scheduled: 3 Current Number of Nodes Scheduled: 3 Number of Nodes Scheduled with Up-to-date Pods: 3 Number of Nodes Scheduled with Available Pods: 3 Number of Nodes Misscheduled: 0 Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: name=prometheus-exporter tier=monitoring Containers: prometheus: Image: prom/node-exporter Port: 80/TCP Environment: Mounts: Volumes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 28s daemon-set Created pod: prometheus-daemonset-pzfl8 Normal SuccessfulCreate 28s daemon-set Created pod: prometheus-daemonset-sjcgh Normal SuccessfulCreate 28s daemon-set Created pod: prometheus-daemonset-ctrg4 ``` Get pods in the DaemonSet: ``` $ kubectl get pods -lname=prometheus-exporter NAME READY STATUS RESTARTS AGE prometheus-daemonset-ctrg4 1/1 Running 0 57s prometheus-daemonset-pzfl8 1/1 Running 0 57s prometheus-daemonset-sjcgh 1/1 Running 0 57s ``` === Limit DaemonSets to specific nodes Verify that the Prometheus pod was successfully deployed to the cluster nodes: $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE prometheus-daemonset-sjcgh 1/1 Running 0 1m 100.96.7.10 ip-172-20-52-200.ec2.internal prometheus-daemonset-ctrg4 1/1 Running 0 1m 100.96.6.10 ip-172-20-64-152.ec2.internal prometheus-daemonset-pzfl8 1/1 Running 0 1m 100.96.5.10 ip-172-20-125-181.ec2.internal Rename one of the node labels as follows: $ kubectl label node ip-172-20-52-200.ec2.internal app=prometheus-node node "ip-172-20-52-200.ec2.internal" labeled Next, edit the DaemonSet template using the command shown: $ kubectl edit ds/prometheus-daemonset Change the `spec.template.spec` to include a `nodeSelector` that matches the changed label: ``` nodeSelector: app: prometheus-node ``` After the update is performed, we have now configured Prometheus to run on a specific node: $ kubectl get ds/prometheus-daemonset NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE prometheus-daemonset 1 1 1 0 1 app=prometheus-node 2m === Delete a DaemonSet Run the following command to delete the DaemonSet: $ kubectl delete -f daemonset.yaml == Job A Job creates one or more pods and ensures that a specified number of them successfully complete. A job keeps track of successful completion of a pod. When the specified number of pods have successfully completed, the job itself is complete. The job will start a new pod if the pod fails or is deleted due to hardware failure. A successful completion of the specified number of pods means the job is complete. This is different from a replica set or a deployment which ensures that a certain number of pods are always running. So if a pod in a replica set or deployment terminates, then it is restarted again. This makes replica set or deployment as long-running processes. This is well suited for a web server, such as NGINX. But a job is completed if the specified number of pods successfully completes. This is well suited for tasks that need to run only once. For example, a job may convert an image format from one to another. Restarting this pod in replication controller would not only cause redundant work but may be harmful in certain cases. Jobs are complementary to Replica Set. A Replica Set manages pods which are not expected to terminate (e.g. web servers), and a Job manages pods that are expected to terminate (e.g. batch jobs). Job is only appropriate for pods with `RestartPolicy` equal to `OnFailure` or `Never`. === Non-parallel Job Only one pod per job is started, unless the pod fails. Job is complete as soon as the pod terminates successfully. Here is the job specification: $ cat job.yaml apiVersion: batch/v1 kind: Job metadata: name: wait spec: template: metadata: name: wait spec: containers: - name: wait image: ubuntu command: ["sleep", "20"] restartPolicy: Never It creates an Ubuntu container, sleeps for 20 seconds and that's it! Create a job using the command: $ kubectl apply -f job.yaml job "wait" created Look at the job: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE wait 1 0 0s The output shows that the job is not successful yet. Watch the pod status to confirm: $ kubectl get -w pods NAME READY STATUS RESTARTS AGE wait-lk49x 1/1 Running 0 7s wait-lk49x 0/1 Completed 0 24s To begin with, it shows that the pod for the job is running. The pod successfully exits after a few seconds and shows the `Completed` status. Now, watch the job status again: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE wait 1 1 1m The output shows that the job was successfully executed. To delete the job, you can run this command $ kubectl delete -f job.yaml === Parallel Job Non-parallel jobs run only one pod per job. This API is used to run multiple pods in parallel for the job. The number of pods to complete is defined by `.spec.completions` attribute in the configuration file. The number of pods to run in parallel is defined by `.spec.parallelism` attribute in the configuration file. The default value for both of these attributes is 1. The job is complete when there is one successful pod for each value in the range in 1 to `.spec.completions`. For that reason, it is also called as _fixed completion count_ job. Here is a job specification: $ cat job-parallel.yaml apiVersion: batch/v1 kind: Job metadata: name: wait spec: completions: 6 parallelism: 2 template: metadata: name: wait spec: containers: - name: wait image: ubuntu command: ["sleep", "20"] restartPolicy: Never This job specification is similar to the non-parallel job specification. It has two new attributes added: `.spec.completions` and `.spec.parallelism`. This means the job will be complete when six pods have successfully completed. A maximum of two pods will run in parallel at a given time. Create a parallel job using the command: $ kubectl apply -f job-parallel.yaml Watch the status of the job as shown: $ kubectl get -w jobs NAME DESIRED SUCCESSFUL AGE wait 6 0 2s wait 6 1 22s wait 6 2 22s wait 6 3 43s wait 6 4 43s wait 6 5 1m wait 6 6 1m The output shows that 2 pods are created about every 20 seconds. In another terminal window, watch the status of pods created: $ kubectl get -w pods -l job-name=wait NAME READY STATUS RESTARTS AGE wait-f7kgb 1/1 Running 0 5s wait-smp4t 1/1 Running 0 5s wait-smp4t 0/1 Completed 0 22s wait-jbdp7 0/1 Pending 0 0s wait-jbdp7 0/1 Pending 0 0s wait-jbdp7 0/1 ContainerCreating 0 0s wait-f7kgb 0/1 Completed 0 22s wait-r5v8n 0/1 Pending 0 0s wait-r5v8n 0/1 Pending 0 0s wait-r5v8n 0/1 ContainerCreating 0 0s wait-r5v8n 1/1 Running 0 1s wait-jbdp7 1/1 Running 0 1s wait-r5v8n 0/1 Completed 0 21s wait-ngrgl 0/1 Pending 0 0s wait-ngrgl 0/1 Pending 0 0s wait-ngrgl 0/1 ContainerCreating 0 0s wait-jbdp7 0/1 Completed 0 21s wait-6l22s 0/1 Pending 0 0s wait-6l22s 0/1 Pending 0 0s wait-6l22s 0/1 ContainerCreating 0 0s wait-ngrgl 1/1 Running 0 1s wait-6l22s 1/1 Running 0 1s wait-ngrgl 0/1 Completed 0 21s wait-6l22s 0/1 Completed 0 21s `kubectl get jobs` shows the status of the job after it has completed: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE wait 6 6 3m Deleting a job deletes all the pods as well. Delete the job as: $ kubectl delete -f job-parallel.yaml == Cron Job === Prerequisites For Kubernetes cluster versions < 1.8, Cron Job can be created with API version `batch/v2alpha1`. You need to explicitly enable API version `batch/v2alpha1` in Kubernetes cluster and perform a rolling-update. If you use *Amazon EKS* for provisioning your Kubernetes cluster, your version should be >= v1.10 and you can proceed without any changes. You can check the cluster version using this command, $ kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-28T20:16:17Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-28T20:13:43Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} === Create Cron Job A Cron Job is a job that runs on a given schedule, written in Cron format. There are two primary use cases: . Run jobs once at a specified point in time . Repeatedly at a specified point in time Here is the job specification: $ cat cronjob.yaml apiVersion: batch/v1beta1 kind: CronJob metadata: name: hello spec: schedule: "*/1 * * * *" jobTemplate: spec: template: metadata: labels: app: hello-cronpod spec: containers: - name: hello image: busybox args: - /bin/sh - -c - date; echo Hello World! restartPolicy: OnFailure This job prints the current timestamp and the message "`Hello World`" every minute. Create the Cron Job as shown in the command: $ kubectl create -f cronjob.yaml Watch the status of the job as shown: $ kubectl get -w cronjobs NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE hello */1 * * * * False 0 hello */1 * * * * False 0 hello */1 * * * * False 1 Tue, 24 Oct 2017 15:41:00 -0700 hello */1 * * * * False 0 Tue, 24 Oct 2017 15:41:00 -0700 hello */1 * * * * False 1 Tue, 24 Oct 2017 15:42:00 -0700 hello */1 * * * * False 0 Tue, 24 Oct 2017 15:42:00 -0700 In another terminal window, watch the status of pods created: $ kubectl get -w pods -l app=hello-cronpod NAME READY STATUS RESTARTS AGE hello-1508884860-cq004 0/1 Pending 0 0s hello-1508884860-cq004 0/1 Pending 0 0s hello-1508884860-cq004 0/1 ContainerCreating 0 0s hello-1508884860-cq004 0/1 Completed 0 1s hello-1508884920-wl5bx 0/1 Pending 0 0s hello-1508884920-wl5bx 0/1 Pending 0 0s hello-1508884920-wl5bx 0/1 ContainerCreating 0 0s hello-1508884920-wl5bx 0/1 Completed 0 2s hello-1508884980-45ktd 0/1 Pending 0 0s hello-1508884980-45ktd 0/1 Pending 0 0s hello-1508884980-45ktd 0/1 ContainerCreating 0 0s hello-1508884980-45ktd 0/1 Completed 0 2s Get logs from one of the pods: $ kubectl logs hello-1508884860-cq004 Tue Oct 24 22:41:02 UTC 2017 Hello World! === Delete Cron Job Delete the Cron Job as shown in the following command: $ kubectl delete -f cronjob.yaml cronjob "hello" deleted == Namespaces Namespaces allows a physical cluster to be shared by multiple teams. A namespace allows to partition created resources into a logically named group. Each namespace provides: . a *unique scope* for resources to avoid name collisions . *policies* to ensure appropriate authority to trusted users . ability to specify *constraints for resource consumption* This allows a Kubernetes cluster to share resources by multiple groups and provide different levels of QoS each group. Resources created in one namespace are hidden from other namespaces. Multiple namespaces can be created, each potentially with different constraints. === Default namespace The list of namespaces can be displayed using the command: $ kubectl get namespace NAME STATUS AGE default Active 2m kube-public Active 2m kube-system Active 2m By default, all resources in Kubernetes cluster are created in a `default` namespace. `kube-public` is the namespace that is readable by all users, even those not authenticated. Any clusters booted with `kubeadm` will have a `cluster-info` ConfigMap. The clusters in this workshop are created using kops and so this ConfigMap will not exist. `kube-system` is the namespace for objects created by the Kubernetes system. Let's create a Deployment: $ kubectl apply -f deployment.yaml deployment "nginx-deployment" created Check its namespace: $ kubectl get deployment -o jsonpath={.items[].metadata.namespace} default === Custom namespace A new namespace can be created using a configuration file or `kubectl`. . The following configuration file can be used to create Namespace: $ cat namespace.yaml kind: Namespace apiVersion: v1 metadata: name: dev labels: name: dev . Create a new Namespace: $ kubectl apply -f namespace.yaml namespace "dev" created . Get the list of Namespaces: $ kubectl get ns NAME STATUS AGE default Active 3h dev Active 12s kube-public Active 3h kube-system Active 3h . Get more details about the Namespace: + ``` $ kubectl describe ns/dev Name: dev Labels: name=dev Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"name":"dev"},"name":"dev","namespace":""}} Status: Active No resource quota. No resource limits. ``` + . Create a Deployment in this new Namespace using a configuration file: + $ cat deployment-namespace.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-deployment-ns namespace: dev spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.12.1 ports: - containerPort: 80 - containerPort: 443 + The main change is the addition of `namespace: dev`. + . Create the Deployment: $ kubectl apply -f deployment-namespace.yaml deployment "nginx-deployment-ns" created . Deployment in a Namespace can be queried by providing an additional switch `-n` as shown: $ kubectl get deployments -n dev NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment-ns 3 3 3 3 1m . Query the Namespace for this Deployment: $ kubectl get deployments/nginx-deployment-ns -n dev -o jsonpath={.metadata.namespace} dev Alternatively, a namespace can be created using `kubectl` as well. . Create a Namespace: $ kubectl create ns dev2 namespace "dev2" created . Create a Deployment: $ kubectl -n dev2 apply -f deployment.yaml deployment "nginx-deployment" created . Get Deployments in the newly created Namespace: $ kubectl get deployments -n dev2 NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 3 3 3 3 1m . Get Deployments in all Namespaces: $ kubectl get deployments --all-namespaces NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default nginx-deployment 3 3 3 3 1h dev nginx-deployment-ns 3 3 3 3 1h dev2 nginx-deployment 3 3 3 3 1m kube-system dns-controller 1 1 1 1 5h kube-system kube-dns 2 2 2 2 5h kube-system kube-dns-autoscaler 1 1 1 1 5h === Quota and Limits Each namespace can be assigned resource quota. Specifying quota allows to restrict how much of cluster resources can be consumed across all resources in a namespace. Resource quota can be defined by a ResourceQuota object. A presence of ResourceQuota object in a namespace ensures that resource quotas are enforced. There can be at most one ResourceQuota object in a namespace. Currently, multiple ResourceQuota objects are allowed. This is filed as https://github.com/kubernetes/kubernetes/issues/55430[kubernetes#55430]. A quota can be specified for compute resources such as CPU and memory, storage resources such as PersistentVolume and PersistentVolumeClaim and number of objects of a given type. A complete list of resources that can be restricted using ResourceQuota are listed at https://kubernetes.io/docs/concepts/policy/resource-quotas/. ==== Create ResourceQuota A ResourceQuota can be created using a configuration file or `kubectl`. . The following configuration file can be used to create ResourceQuota: $ cat resource-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: quota spec: hard: cpu: "4" memory: 6G pods: "10" replicationcontrollers: "3" services: "5" configmaps: "5" + This configuration file places the following requirements on the namespace: + .. Every new Container created must have a memory and CPU limit .. Total number of Pods in this namespace cannot exceed 10 .. Total number of ReplicationController in this namespace cannot exceed 3 .. Total number of Service in this namespace cannot exceed 5 .. Total number of ConfigMap in this namespace cannot exceed 5 + . Create a new ResourceQuota: $ kubectl apply -f resource-quota.yaml resourcequota "quota" created + Alternatively, a ResourceQuota may be created using the `kubectl` CLI: + kubectl create resourcequota quota2 --hard=cpu=10,memory=6G,pods=10,services=5,replicationcontrollers=3 + In either this case, these restrictions would be placed on the `default` namespace in this case. An alternate namespace can be specified either in the configuration file or using the `--namespace` option on the `kubectl` CLI. + . Get the list of ResourceQuota: $ kubectl get quota NAME AGE quota 25s . Get more details about the ResourceQuota: $ kubectl describe quota/quota Name: quota Namespace: default Resource Used Hard -------- ---- ---- configmaps 0 5 cpu 300m 4 memory 0 6G pods 3 10 replicationcontrollers 0 3 services 1 5 + The output shows that three Pods and one Service already exists in the `default` namespace. ==== Scale resources with ResourceQuota Now that the ResourceQuota has been created, let's see how this impacts the new resources that are created or existing resources that are scaled. We already have a Deployment `nginx-deployment`. Let's scale the number of replicas to exceed the assigned quota and see what happens. . Scale the number of replicas for the Deployment: $ kubectl scale --replicas=12 deployment/nginx-deployment deployment "nginx-deployment" scaled + The command output says that the Deployment is scaled. + . Let's check if all the replicas are available: + $ kubectl get deployment/nginx-deployment -o jsonpath={.status.availableReplicas} 3 + It shows only three replicas are available. + . More details can be found: + $ kubectl describe deployment nginx-deployment ... Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available False MinimumReplicasUnavailable ReplicaFailure True FailedCreate + The current reason is displayed in the output. ==== Create resources with ResourceQuota Let's create a Pod with the following configuration file: $ cat pod.yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod labels: name: nginx-pod spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 You may have to remove a previously running Pod or Deployment before attempting to create this Pod. $ kubectl apply -f pod.yaml Error from server (Forbidden): error when creating "pod.yaml": pods "nginx-pod" is forbidden: failed quota: quota: must specify memory The error message indicates that a ResourceQuota is in effect, and that the Pod must explicitly specify memory resources. Update the configuration file to: $ cat pod-cpu-memory.yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod labels: name: nginx-pod spec: containers: - name: nginx image: nginx:latest resources: requests: memory: "100m" ports: - containerPort: 80 There is an explicity memory resource defined here. Now, try to create the pod: $ kubectl apply -f pod-cpu-memory.yaml pod "nginx-pod" created The Pod is successfully created. Get more details about the Pod: $ kubectl get pod/nginx-pod -o jsonpath={.spec.containers[].resources} map[requests:map[cpu:1 memory:100m] Get more details about the ResourceQuota: $ kubectl describe quota/quota Name: quota Namespace: default Resource Used Hard -------- ---- ---- configmaps 0 5 cpu 400m 4 memory 100m 6G pods 4 12 replicationcontrollers 0 3 services 1 5 Note, how CPU and memory resources have incremented values. https://github.com/kubernetes/kubernetes/issues/55433[kubernetes#55433] provide more details on how an explicit CPU resource is not needed to create a Pod with ResourceQuota. $ kubectl delete quota/quota $ kubectl delete quota/quota2 You are now ready to continue on with the workshop! :frame: none :grid: none :valign: top [align="center", cols="3", grid="none", frame="none"] |===== |image:button-continue-standard.png[link=../../02-path-working-with-clusters/201-cluster-monitoring] |image:button-continue-developer.png[link=../../03-path-application-development/301-local-development] |image:button-continue-operations.png[link=../../02-path-working-with-clusters/201-cluster-monitoring] |link:../../standard-path.adoc[Go to Standard Index] |link:../../developer-path.adoc[Go to Developer Index] |link:../../operations-path.adoc[Go to Operations Index] |=====