= Kubernetes Cluster Auto-scaling
:toc:
:icons:
:linkcss:
:imagesdir: ../../resources/images
This tutorial will walk you through how to setup cluster scaling using `cluster-autoscaler` to enable worker node autoscaling based on multiple metrics within Kubernetes.

== Prerequisites

In order to perform exercises in this chapter, you’ll need to deploy configurations to a Kubernetes cluster. To create an EKS-based Kubernetes cluster, use the link:../../01-path-basics/102-your-first-cluster#create-a-kubernetes-cluster-with-eks[AWS CLI] (recommended). If you wish to create a Kubernetes cluster without EKS, you can instead use link:../../01-path-basics/102-your-first-cluster#alternative-create-a-kubernetes-cluster-with-kops[kops].

All configuration files for this chapter are in the `cluster-scaling` directory. Make sure you change to that directory before giving any commands in this chapter.

=== Autoscaling group size

Check the size of your worker node autoscaling group and make sure the MaxSize is greater than 2. You can do this in the AWS Console, or use the CLI command below

    aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names nodes.<cluster-name>

For example
```
$ aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names nodes.cluster.k8s.local
{
    "AutoScalingGroups": [
        {
            "AutoScalingGroupARN": "arn:aws:autoscaling:us-east-1:<account-id>:autoScalingGroup:xxx:autoScalingGroupName/nodes.cluster.k8s.local",
            .
            .
            "MaxSize": 10,
}
```

The exact value can be identified using the command:

  aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names nodes.cluster.k8s.local | jq .AutoScalingGroups[0].MaxSize

If the value of `MaxSize` is 2 (which means you can't scale beyond 2 worker nodes) then update it using kops using the following command:

    kops edit ig nodes

It opens up the cluster configuration file as shown:

Using cluster from kubectl context: example.cluster.k8s.local

  apiVersion: kops/v1alpha2
  kind: InstanceGroup
  metadata:
    creationTimestamp: 2017-10-29T01:02:55Z
    labels:
      kops.k8s.io/cluster: example.cluster.k8s.local
    name: nodes
  spec:
    image: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-07-28
    machineType: t2.medium
    maxSize: 5
    minSize: 5
    role: Node
    subnets:
    - eu-central-1a
    - eu-central-1b
    - eu-central-1c

Change the value of `maxSize` to `10`. Save the configuration file and exit out of the editor.

Update cluster configuration:

    kops update cluster --name example.cluster.k8s.local --yes

It shows output as:

```
$ kops update cluster --name example.cluster.k8s.local --yes
I1029 14:09:37.853640   15831 apply_cluster.go:420] Gossip DNS: skipping DNS validation
I1029 14:09:38.925971   15831 executor.go:91] Tasks: 0 done / 81 total; 36 can run
I1029 14:09:40.200162   15831 executor.go:91] Tasks: 36 done / 81 total; 15 can run
I1029 14:09:41.293803   15831 executor.go:91] Tasks: 51 done / 81 total; 22 can run
I1029 14:09:43.034786   15831 executor.go:91] Tasks: 73 done / 81 total; 5 can run
I1029 14:09:43.868003   15831 executor.go:91] Tasks: 78 done / 81 total; 3 can run
I1029 14:09:44.655604   15831 executor.go:91] Tasks: 81 done / 81 total; 0 can run
I1029 14:09:44.991551   15831 update_cluster.go:247] Exporting kubecfg for cluster
kops has set your kubectl context to example.cluster.k8s.local

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster
```

Check that there is no need to do a rolling update of the cluster:

```
$ kops rolling-update cluster
Using cluster from kubectl context: example.cluster.k8s.local

NAME                  STATUS  NEEDUPDATE  READY MIN MAX NODES
master-eu-central-1a  Ready   0           1     1   1   1
master-eu-central-1b  Ready   0           1     1   1   1
master-eu-central-1c  Ready   0           1     1   1   1
nodes                 Ready   0           5     5   10  5

No rolling-update required.
```

=== IAM permissions

Each worker running in your Kubernetes cluster MUST have the necessary IAM Policy attached. If you previously set up your cluster using `kops` you can modify the existing permissions on your worker IAM Role or you can manually create a new policy and attach it to the IAM Role. Both methods are shown below.

=== Using kops to update the IAM policy

Use kops to edit the cluster configuration

    kops edit cluster --name example.cluster.k8s.local

This opens up an editor showing the configuration of the cluster. Add the `additionalPolicies` section below to the end of the Cluster spec; your spec should look something like this

```
spec:
.
.
  topology:
    dns:
      type: Public
    masters: public
    nodes: public
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
          ],
          "Resource": ["*"]
        }
      ]
```

Note, the first few lines are shown for continuity.

Update the cluster:

```
$ kops update cluster --name example.cluster.k8s.local --yes
I1029 15:25:24.068325   21411 apply_cluster.go:420] Gossip DNS: skipping DNS validation
I1029 15:25:25.002684   21411 executor.go:91] Tasks: 0 done / 81 total; 36 can run
I1029 15:25:26.359336   21411 executor.go:91] Tasks: 36 done / 81 total; 15 can run
I1029 15:25:27.378808   21411 executor.go:91] Tasks: 51 done / 81 total; 22 can run
I1029 15:25:29.512767   21411 executor.go:91] Tasks: 73 done / 81 total; 5 can run
I1029 15:25:30.338608   21411 executor.go:91] Tasks: 78 done / 81 total; 3 can run
I1029 15:25:31.189236   21411 executor.go:91] Tasks: 81 done / 81 total; 0 can run
I1029 15:25:31.586799   21411 update_cluster.go:247] Exporting kubecfg for cluster
kops has set your kubectl context to example.cluster.k8s.local

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster
```

There is no need to rolling update the cluster.

=== Manually attaching the IAM policy to the role

The policy below must be attached to the role assigned to the Kubernetes worker nodes. The role definition exists in the file templates/asg-policy.json

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeAutoScalingInstances",
          "autoscaling:SetDesiredCapacity",
          "autoscaling:TerminateInstanceInAutoScalingGroup"
        ],
        "Resource": "*"
      }
    ]
  }

To configure these permissions, you need to create the policy using the command below.

    aws iam create-policy --policy-document file://templates/asg-policy.json --policy-name ClusterAutoScaling

You will see a response similar to this:

```
  $ aws iam create-policy --policy-document file://templates/asg-policy.json --policy-name ClusterAutoScaling
  => {
    "Policy": {
        "PolicyName": "ClusterAutoScaling",
        "PolicyId": "ANPAJVCFZ6I4OL6BGFGD2",
        "Arn": "arn:aws:iam::<account-id>:policy/ClusterAutoScaling",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "IsAttachable": true,
        "CreateDate": "2017-10-05T20:35:54.964Z",
        "UpdateDate": "2017-10-05T20:35:54.964Z"
    }
  }
```

Then attach the policy to the role assigned to the Kubernetes worker nodes. To attach the policy to the IAM Role, you first need to get the name of the role; if you set up your cluster using `kops`, this will be `nodes.[DOMAIN]` such as `nodes.cluster.k8s.local`

From the output of the `create-policy` command get the `.Policy.Arn` attribute and use that to add the policy to the role. Alternatively, you can use this convenience command which retrieves your AWS Account ID using AWS CLI:

    aws iam attach-role-policy --role-name nodes.cluster.k8s.local --policy-arn arn:aws:iam::`aws sts get-caller-identity --output text --query 'Account'`:policy/ClusterAutoScaling

== Deploy Worker Autoscaler

Before running the command below, update the following attributes in file `templates/2-10-autoscaler.yaml`:

. `command  --nodes` to the name of your nodes ASG
. `env.value` to the name of your region

You can find the name of nodes ASG using this command

  $ aws autoscaling describe-auto-scaling-groups --query 'AutoScalingGroups[].AutoScalingGroupName'
  [
      "master-eu-central-1a.masters.cluster.k8s.local",
      "master-eu-central-1b.masters.cluster.k8s.local",
      "master-eu-central-1c.masters.cluster.k8s.local",
      "nodes.example.cluster.k8s.local"
  ]

The last value in this output is the name of the nodes ASG. If the default cluster name of `example.cluster.k8s.local` was used to create the cluster, then there is no need to make any changes to the configuration file.

Now, install the `cluster-autoscaler` with a configuration of `min: 2, max: 10, name: cluster-autoscaler`

  $ kubectl apply -f templates/2-10-autoscaler.yaml
  deployment "cluster-autoscaler" created

Once this is deployed you can view the logs by running

  kubectl logs deployment/cluster-autoscaler --namespace=kube-system

The output will be shown as:

```
I1029 22:49:19.880269       1 main.go:225] Cluster Autoscaler 0.6.0
I1029 22:49:19.995396       1 leaderelection.go:179] attempting to acquire leader lease...
I1029 22:49:20.075665       1 leaderelection.go:189] successfully acquired lease kube-system/cluster-autoscaler
I1029 22:49:20.075796       1 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cluster-autoscaler", UID:"6677810d-bcfb-11e7-a483-0681c180117e", APIVersion:"v1", ResourceVersion:"140681", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster-autoscaler-33142225-z150r became leader
I1029 22:49:20.076730       1 reflector.go:198] Starting reflector *v1.Pod (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:144

. . .

I1029 22:50:21.488144       1 cluster.go:89] Fast evaluation: node ip-172-20-109-10.eu-central-1.compute.internal cannot be removed: non-daemonset, non-mirrored, non-pdb-assigned kube-system pod present: kube-dns-autoscaler-4184363331-jh7jb
I1029 22:50:21.488152       1 cluster.go:75] Fast evaluation: ip-172-20-75-132.eu-central-1.compute.internal for removal
I1029 22:50:21.488172       1 cluster.go:89] Fast evaluation: node ip-172-20-75-132.eu-central-1.compute.internal cannot be removed: non-daemonset, non-mirrored, non-pdb-assigned kube-system pod present: kube-dns-729475360-z4d1r
I1029 22:50:23.324479       1 leaderelection.go:204] successfully renewed lease kube-system/cluster-autoscaler
```

== Validation

To validate that the `cluster-autoscaler` is properly working you can use the `aws` CLI to request the current `DesiredCapacity` of your ASG with

  export ASG_NAME=nodes.example.cluster.k8s.local
  aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names=$ASG_NAME --query 'AutoScalingGroups[0].DesiredCapacity'

You should see a result of 5, or whatever was the initial size of your cluster.

Check the max size of your cluster:

  $ aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names=$ASG_NAME --query 'AutoScalingGroups[0].MaxSize'
  10

This correctly shows 10, as was set earlier in this chapter.

Then you can deploy an application which requests more resources than your cluster has available see `templates/dummy-resource-offers.yaml` for reference.

NOTE: Depending on the size of your cluster this might not trigger autoscaling. Increase the `replicas: 10` count to the necessary amount you need to fill your clusters resources.

  $ kubectl apply -f templates/dummy-resource-offers.yaml
  service "greeter" created
  deployment "greeter" created

After this loads you can use the `describe-auto-scaling-groups` command again to see the `DesiredCapacity` change.

  aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names=$ASG_NAME --query 'AutoScalingGroups[0].DesiredCapacity'


If you have deployed Heapster, as described in the link:../cluster-monitoring/README.adoc#heapster-influxdb-and-grafana[Cluster Monitoring] lab, you can use this command to see the resource usage of your nodes:

```
$ kubectl top nodes
NAME                                             CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
ip-172-20-109-10.eu-central-1.compute.internal   543m         27%       1722Mi          44%
ip-172-20-44-33.eu-central-1.compute.internal    125m         12%       2120Mi          57%
ip-172-20-75-132.eu-central-1.compute.internal   607m         30%       1733Mi          44%
ip-172-20-41-77.eu-central-1.compute.internal    450m         22%       1703Mi          44%
ip-172-20-85-128.eu-central-1.compute.internal   86m          8%        2049Mi          55%
ip-172-20-93-108.eu-central-1.compute.internal   534m         26%       1747Mi          45%
ip-172-20-106-93.eu-central-1.compute.internal   522m         26%       1734Mi          44%
ip-172-20-101-20.eu-central-1.compute.internal   101m         5%        2046Mi          55%
```

Once auto-scaling triggers, you should see a result of a higher number of nodes than original; this may take a few minutes:

  $ aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names=$ASG_NAME --query '.AutoScalingGroups[0].DesiredCapacity'
  5

If auto-scaling does not trigger, then you can increase the number of replicas using the command:

  $ kubectl scale --replicas=30 deployment/greeter
  deployment "greeter" scaled

Now, auto-scaling may trigger, based upon your cluster configuration. The updated query for ASG may look like as shown:

  $ aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names=$ASG_NAME --query 'AutoScalingGroups[0].DesiredCapacity'
  10

It takes a few minutes for the additional worker nodes to start and become part of the cluster. The updated nodes information is now shown:

```
$ kubectl top nodes
NAME                                              CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
ip-172-20-85-128.eu-central-1.compute.internal    74m          7%        2088Mi          56%
ip-172-20-93-108.eu-central-1.compute.internal    25m          1%        1734Mi          44%
ip-172-20-109-10.eu-central-1.compute.internal    26m          1%        1716Mi          44%
ip-172-20-86-51.eu-central-1.compute.internal     24m          1%        1075Mi          27%
ip-172-20-51-221.eu-central-1.compute.internal    21m          1%        1074Mi          27%
ip-172-20-61-253.eu-central-1.compute.internal    22m          1%        1075Mi          27%
ip-172-20-41-77.eu-central-1.compute.internal     31m          1%        1716Mi          44%
ip-172-20-106-93.eu-central-1.compute.internal    27m          1%        1745Mi          45%
ip-172-20-101-20.eu-central-1.compute.internal    94m          4%        2078Mi          56%
ip-172-20-44-33.eu-central-1.compute.internal     112m         11%       2148Mi          58%
ip-172-20-116-218.eu-central-1.compute.internal   22m          1%        1070Mi          27%
ip-172-20-44-50.eu-central-1.compute.internal     18m          0%        1076Mi          27%
ip-172-20-75-132.eu-central-1.compute.internal    27m          1%        1723Mi          44%
```

You are now ready to continue on with the workshop!

:frame: none
:grid: none
:valign: top

[align="center", cols="1", grid="none", frame="none"]
|=====
|image:button-continue-operations.png[link=../../02-path-working-with-clusters/205-cluster-autoscaling]
|link:../../operations-path.adoc[Go to Operations Index]
|=====
