= Application Configuration and Secrets
:toc:
:icons:
:linkcss:
:imagesdir: ../../resources/images

Separating application code from configuration is one of the factors when building an application using https://12factor.net/[12-factor]. This allows the same application to be deployed across multiple environments, such as dev, test, staging and production. It also allows the application to be more portable.

Kubernetes has native constructs like _ConfigMap_ and _Secrets_ that allow you to decouple configuration artifacts from the image content to keep containerized applications portable. In addition, other services as https://aws.amazon.com/ec2/systems-manager/parameter-store/[AWS Parameter Store] or https://www.vaultproject.io/[Hashicorp Vault] can be used to store that information as well.

This chapter will cover how these constructs and services can be used to store configuration information and secrets.

. ConfigMap is just a set of key-value pairs. It allow you to decouple configuration artifacts from image content.
. Secrets allows separating sensitive information such as credentials and keys from an application.

ConfigMap is similar to Secrets, but provides a means of working with strings that don’t contain sensitive information.

Make sure you change to that directory before giving any commands in this chapter.

== Prerequisites

In order to perform exercises in this chapter, you’ll need to deploy configurations to a Kubernetes cluster. To create an EKS-based Kubernetes cluster, use the link:../../01-path-basics/102-your-first-cluster#create-a-kubernetes-cluster-with-eks[AWS CLI] (recommended). If you wish to create a Kubernetes cluster without EKS, you can instead use link:../../01-path-basics/102-your-first-cluster#alternative-create-a-kubernetes-cluster-with-kops[kops].

All configuration files for this chapter are in the `config-secrets` directory.

== Configuration data using Kubernetes ConfigMap

This section will explain:

. Pass configuration information to a Pod
. Define environment variables in a Pod using ConfigMap

=== Create a ConfigMap object

Create a ConfigMap:

    $ kubectl apply -f ./templates/redis-configmap.yaml
    configmap "redis-config" created

`redis-configmap.yaml` is a standard resource configuration file. It defines the configuration information as:

    data:
      redis-config: |
        maxmemory=2mb
        maxmemory-policy=allkeys-lru

The configuration data is stored in the main key `data`. `redis-config` is an attribute inside this key where the configuration information for the Redis pod is defined as key-value pairs.

Get the list of ConfigMaps:

    $ kubectl get configmap
    NAME           DATA      AGE
    redis-config   1         14s

Get more details about the created ConfigMap:

```
$ kubectl get configmap/redis-config -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    redis-config: |
      maxmemory 2mb
      maxmemory-policy allkeys-lru
  kind: ConfigMap
  metadata:
    creationTimestamp: 2017-10-22T18:38:27Z
    labels:
      k8s-app: redis
    name: redis-config
    namespace: default
    resourceVersion: "302238"
    selfLink: /api/v1/namespaces/default/configmaps/redis-config
    uid: 316309d0-b758-11e7-8c3f-06329c8974cc
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
```

The configuration information is shown as key/value pairs in the `data` key.

==== Alternative ways to create ConfigMap

We created a ConfigMap using a resource configuration file. Other ways to create ConfigMap are listed below:

NOTE: These ConfigMaps are using the exact same name as the one previously created. If you like to try the commands, then you either need to give a different name to the ConfigMap or delete the previously created ConfigMap using the command `kubectl delete -f ./templates/redis-configmap.yaml`.

. `kubectl create configmap --from-literal=<key>:<value>`. Multiple `--from-literal=<key>:<value>` options can be used to define different key/value pairs. For example:

  $ kubectl create configmap redis-config --from-literal=maxmemory=2mb --from-literal=maxmemory-policy=allkeys-lru
  configmap "redis-config" created
+
More details about the ConfigMap can be obtained as:
+
  $ kubectl get configmap/redis-config -o yaml
  apiVersion: v1
  data:
    maxmemory: 2mb
    maxmemory-policy: allkeys-lru
  kind: ConfigMap
  metadata:
    creationTimestamp: 2017-10-22T15:29:31Z
    name: redis-config
    namespace: default
    resourceVersion: "287452"
    selfLink: /api/v1/namespaces/default/configmaps/redis-config
    uid: cccf20b7-b73d-11e7-8c3f-06329c8974cc
+
. `kubectl create configmap redis-config --from-file=<properties file>` where `<properties file>` is a property file with key/value pairs. For example, `templates/redis-config` looks like:
+
  maxmemory 2mb
  maxmemory-policy allkeys-lru
+
And now the ConfigMap can be created as:
+
  $ kubectl create configmap redis-config --from-file=templates/redis-config
  configmap "redis-config" created
+
More details about the ConfigMap can be obtained as:
+
  $ kubectl get configmap/redis-config -o yaml
  apiVersion: v1
  data:
    redis-config: |
      maxmemory=2mb
      maxmemory-policy=allkeys-lru
  kind: ConfigMap
  metadata:
    creationTimestamp: 2017-10-22T15:56:08Z
    name: redis-config
    namespace: default
    resourceVersion: "289533"
    selfLink: /api/v1/namespaces/default/configmaps/redis-config
    uid: 84901162-b741-11e7-8c3f-06329c8974cc
+
The filename becomes a key stored in the data section of the ConfigMap. The file contents become the key’s value.

At the end of this section, you'll have created a ConfigMap `redis-config`.

=== Consume in a pod volume

A ConfigMap must be created before referencing it in a Pod specification (unless you mark the ConfigMap as "`optional`"). If you reference a ConfigMap that doesn’t exist would , the Pod won’t start.

Let's use `redis-config` ConfigMap to create our `redis.conf` configuration file in the pod `redis-pod`. It maps the ConfigMap to the volume where the configuration resides:

    $ kubectl apply -f ./templates/redis-pod.yaml
    pod "redis-pod" created

Wait for the pod to run:

    $ kubectl get pods
    NAME        READY     STATUS    RESTARTS   AGE
    redis-pod   1/1       Running   0          12m

Check logs from the pod to verify that Redis has started:

  $ kubectl logs redis-pod
                  _._
             _.-``__ ''-._
        _.-``    `.  `_.  ''-._           Redis 2.8.19 (00000000/0) 64 bit
    .-`` .-```.  ```\/    _.,_ ''-._
   (    '      ,       .-`  | `,    )     Running in stand alone mode
   |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
   |    `-._   `._    /     _.-'    |     PID: 6
    `-._    `-._  `-./  _.-'    _.-'
   |`-._`-._    `-.__.-'    _.-'_.-'|
   |    `-._`-._        _.-'_.-'    |           http://redis.io
    `-._    `-._`-.__.-'_.-'    _.-'
   |`-._`-._    `-.__.-'    _.-'_.-'|
   |    `-._`-._        _.-'_.-'    |
    `-._    `-._`-.__.-'_.-'    _.-'
        `-._    `-.__.-'    _.-'
            `-._        _.-'
                `-.__.-'

  [6] 22 Oct 18:39:45.386 # Server started, Redis version 2.8.19
  [6] 22 Oct 18:39:45.386 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
  [6] 22 Oct 18:39:45.386 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
  [6] 22 Oct 18:39:45.386 * The server is now ready to accept connections on port 6379

Validate that your redis cluster picked up the appropriate configuration:

    $ kubectl exec redis-pod -it redis-cli
    127.0.0.1:6379> CONFIG GET maxmemory
    1) "maxmemory"
    2) "2097152"
    127.0.0.1:6379> CONFIG GET maxmemory-policy
    1) "maxmemory-policy"
    2) "allkeys-lru"
    127.0.0.1:6379> quit

You should see the same values that were specified in `./templates/redis-configmap.yaml` outputted in the above commands.

Now, changing the pod configuration would involve the following steps:

. Edit `redis-configmap.yaml`
. Update the ConfigMap using the command: `kubectl apply -f templates/redis-configmap.yaml`
. Wrap the pod in a Deployment
. Terminate the pod, Deployment will restart the pod and pick up new configuration

=== Consume as pod environment variables

The data from ConfigMap can be used to initialize environment variables in a pod. We'll use `arungupta/print-hello` image to print "`Hello World`" on the console. The number of times this message is printed is defined by an environment variable `COUNT`. This value of this variable is defined in the ConfigMap.

==== Create a pod and use ConfigMap

. Create a ConfigMap:

  $ kubectl create configmap hello-config --from-literal=COUNT=2
  configmap "hello-config" created

. Get more details about this ConfigMap:

  $ kubectl get configmap/hello-config -o yaml
  apiVersion: v1
  data:
    COUNT: "2"
  kind: ConfigMap
  metadata:
    creationTimestamp: 2017-10-26T21:40:10Z
    name: hello-config
    namespace: default
    resourceVersion: "92516"
    selfLink: /api/v1/namespaces/default/configmaps/hello-config
    uid: 3dacb22f-ba96-11e7-ab9c-123f969a2ce2

. Use this ConfigMap to create a pod:

  $ kubectl apply -f templates/app-pod.yaml
  pod "app-pod" created
+
The pod configuration file looks like:
+
  apiVersion: v1
  kind: Pod
  metadata:
    labels:
      name: app-pod
    name: app-pod
  spec:
    containers:
    - name: app
      image: arungupta/print-hello:latest
      env:
      - name: COUNT
        valueFrom:
          configMapKeyRef:
            name: hello-config
            key: COUNT
      ports:
      - containerPort: 8080

. Observe logs from the pod:

  $ kubectl logs -f app-pod
  npm info it worked if it ends with ok
  npm info using npm@3.10.10
  npm info using node@v6.11.4
  npm info lifecycle webapp@1.0.0~prestart: webapp@1.0.0
  npm info lifecycle webapp@1.0.0~start: webapp@1.0.0

  > webapp@1.0.0 start /usr/src/app
  > node server.js

  Running on http://0.0.0.0:8080

. In a new terminal, expose the pod as a Service:

  $ kubectl expose pod app-pod --port=80 --target-port=8080 --name=app
  service "app" exposed

. Start Kubernetes proxy:

  kubectl proxy --address 0.0.0.0 --accept-hosts '.*' --port 8080

. In a new terminal, access the service as:

  $ curl -k https://ENVIRONMENT_ID.vfs.cloud9.REGION_ID.amazonaws.com/api/v1/proxy/namespaces/default/services/app/
  printed 2 times
+
The pod logs are refreshed as well:
+
  Hello world 0
  Hello world 1

==== Change the ConfigMap and verify pod logs

. Edit the ConfigMap:

  $ kubectl edit configmap/hello-config

. Change the value to `4`
. Terminate the pod:

  $ kubectl delete pod/app-pod
  pod "app-pod" deleted

. Run the pod again:

  kubectl create -f templates/app-pod.yaml
  pod "app-pod" created

. Access the service again:

  $ curl -k https://ENVIRONMENT_ID.vfs.cloud9.REGION_ID.amazonaws.com/api/v1/proxy/namespaces/default/services/app/
  printed 4 times

. Logs from the pod are refreshed:

  Hello world 0
  Hello world 1
  Hello world 2
  Hello world 3

== Secrets using Kubernetes Secrets

In this section we will demonstrate how to place secrets into the Kubernetes cluster and then show multiple ways of retrieving those secretes from within a pod.

=== Create secrets

First encode the secrets you want to apply, for this example we will use the username `admin` and the password `password`

    echo -n "admin" | base64
    echo -n "password" | base64

Both of these values are already written in the file `./templates/secret.yaml`. The configuration looks like:

```
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQ=
```

You can now insert this secret in the Kubernetes cluster with the following command:

  kubectl apply -f ./templates/secret.yaml

The list of created secrets can be seen as:

  $ kubectl get secrets
  NAME                  TYPE                                  DATA      AGE
  default-token-4cqsx   kubernetes.io/service-account-token   3         8h
  mysecret              Opaque                                2         6s

The values of the secret are displayed as `Opaque`.

Get more details about the secret:

  $ kubectl describe secrets/mysecret
  Name:         mysecret
  Namespace:    default
  Labels:       <none>
  Annotations:  <none>

  Type:  Opaque

  Data
  ====
  password:  8 bytes
  username:  5 bytes

Once again, the values of the secret are not shown.

=== Consume in a pod volume

Deploy the pod:

    kubectl apply -f ./templates/pod-secret-volume.yaml

The pod configuration file looks like:

    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-secret-volume
    spec:
      containers:
      - name: pod-secret-volume
        image: redis
        volumeMounts:
        - name: foo
          mountPath: "/etc/foo"
          readOnly: true
      volumes:
      - name: foo
        secret:
          secretName: mysecret

Open a shell to the pod to see the secrets:

    kubectl exec -it pod-secret-volume /bin/bash
    ls /etc/foo
    cat /etc/foo/username ; echo
    cat /etc/foo/password ; echo

The above commands should result in the plain text values, the decoding is done for you.

Delete the pod:

    kubectl delete -f ./templates/pod-secret-volume.yaml

=== Consume as pod environment variables

Deploy the pod:

    kubectl apply -f ./templates/pod-secret-env.yaml

The pod configuration file looks like:

    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-secret-env
    spec:
      containers:
      - name: pod-secret-env
        image: redis
        env:
          - name: SECRET_USERNAME
            valueFrom:
              secretKeyRef:
                name: mysecret
                key: username
          - name: SECRET_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysecret
                key: password
      restartPolicy: Never

Open a shell to the pod to see the secrets:

    kubectl exec -it pod-secret-env /bin/bash
    echo $SECRET_USERNAME
    echo $SECRET_PASSWORD

The above commands illustrate how to see the secret values via environment variables.

== Configuration data and Secrets using AWS Parameter Store

https://aws.amazon.com/ec2/systems-manager/[Amazon EC2 Systems Manager] eases the configuration and management of Amazon EC2 instances and associated resources. One of the features of Systems Manager is https://aws.amazon.com/ec2/systems-manager/parameter-store/[Parameter Store] that provides a centralized location to store, provide access control, and easily reference your configuration data, whether plain-text data such as database strings or secrets such as passwords, encrypted through https://aws.amazon.com/kms/[AWS Key Management Service] (KMS).

KMS helps you encrypt your sensitive information and protect the security of your keys. Additionally, all calls to the parameter store are recorded with AWS CloudTrail so that they can be audited. Access to each parameter store secrets can be scoped with IAM.

Parameter Store allows three types of configuration data to be stored:

- String
- List of string
- Secure string

This section will show how to create a secure string using AWS CLI and access it in a Pod.

=== Create KMS Key

. Create a new encryption key: https://console.aws.amazon.com/iam/home#/encryptionKeys/
. Click on `Create key`. If you haven't used the KMS service before, click `Get Started`.
. Specify the `Alias` as `k8s-key`
+
image::aws-kms-create-key.png[]
+
Click on `Next Step`.
. Take the defaults for `Add Tags` and click on `Next Step`.
. Select the IAM user(s) and roles that can administer this key through the KMS API
+
image::aws-kms-key-admins.png[]
+
. Select the IAM user(s) and roles that can use this key to encrypt and decrypt data from within applications. We'll use the IAM role that is assigned to the worker nodes in the Kubernetes cluster created by kops.
+
image::aws-kms-key-usage-perms.png[]
+
. Preview key policy:

  {
    "Id": "key-consolepolicy-3",
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "Enable IAM User Permissions",
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "arn:aws:iam::<account-id>:root"
          ]
        },
        "Action": "kms:*",
        "Resource": "*"
      },
      {
        "Sid": "Allow access for Key Administrators",
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "arn:aws:iam::<account-id>:user/arun",
            "arn:aws:iam::<account-id>:role/nodes.example.cluster.k8s.local"
          ]
        },
        "Action": [
          "kms:Create*",
          "kms:Describe*",
          "kms:Enable*",
          "kms:List*",
          "kms:Put*",
          "kms:Update*",
          "kms:Revoke*",
          "kms:Disable*",
          "kms:Get*",
          "kms:Delete*",
          "kms:TagResource",
          "kms:UntagResource",
          "kms:ScheduleKeyDeletion",
          "kms:CancelKeyDeletion"
        ],
        "Resource": "*"
      },
      {
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "arn:aws:iam::<account-id>:role/nodes.example.cluster.k8s.local"
          ]
        },
        "Action": [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:DescribeKey"
        ],
        "Resource": "*"
      },
      {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "arn:aws:iam::<account-id>:role/nodes.example.cluster.k8s.local"
          ]
        },
        "Action": [
          "kms:CreateGrant",
          "kms:ListGrants",
          "kms:RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
          "Bool": {
            "kms:GrantIsForAWSResource": true
          }
        }
      }
    ]
  }

. Click on `Finish`.
. Select `IAM`, `Encryption Keys`, `k8s-key` and copy the ARN of the key.

=== Update the IAM role

For a Kubernetes cluster created by kops, EC2 worker nodes use an instance profile to allow the EC2 instances to access other AWS services. This role must be updated to allow the worker nodes to read the secrets from Parameter Store.

In the IAM Console click `roles` and type `nodes` into the search box. Find the `nodes.example.cluster.k8s.local` role
and click it. In the Permissions tab, expand the inline policy for `nodes.example.cluster.k8s.local` and click
`Edit policy`. Add the `ssm:GetParameter` permission to the policy so the policy looks similar to the one below.

    {
        "Version": "2012-10-17",
        "Statement": [
            .
            .
            .
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ssm:GetParameter"
                ],
                "Resource": [
                    "arn:aws:ssm:<region>:<account-id>:parameter/GREETING",
                    "arn:aws:ssm:<region>:<account-id>:parameter/NAME"
                ]
            }
        ]
    }

=== Create secrets

Only the value of the secure string parameter is encrypted. The name of the parameter, description, and other properties are not encrypted.

. A secret in AWS Parameter is created as a secure string. Create a secure string:
+
  $ aws ssm put-parameter \
    --name GREETING \
    --value Hello \
    --type SecureString \
    --key-id arn:aws:kms:<region>:<account-id>:key/414a963b-7fe4-4a61-b19f-ea408b9bda3b
  {
      "Version": 1
  }
+
This will create a secret in the Parameter Store using the KMS key.
+
. Get the value of the created secret:
+
```
$ aws ssm get-parameter --name GREETING
{
    "Parameter": {
        "Version": 1,
        "Type": "SecureString",
        "Name": "GREETING",
        "Value": "AQICAHghFIWYznvdUrX6qDhd5xLFHpoaQ5WL1EaHqsbkenfFEwHdqTpU8URwKMf2H9XmMyMgAAAAYzBhBgkqhkiG9w0BBwagVDBSAgEAME0GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM0jZaUadELhmiCzj4AgEQgCBXVAZzfjac8P2AFrelnLaXb3z7ssZt2q/npxYAdJ9ABQ=="
    }
}
```
+
By default, the encrypted value of the secret is shown in the output.
+
. Decrypted value of the secret can be obtained:

  $ aws ssm get-parameter --name GREETING --with-decryption
  {
      "Parameter": {
          "Version": 1,
          "Type": "SecureString",
          "Name": "GREETING",
          "Value": "Hello"
      }
  }

. Create another secret:

  $ aws ssm put-parameter \
    --name NAME \
    --value World \
    --type SecureString \
    --key-id arn:aws:kms:<region>:<account-id>:key/414a963b-7fe4-4a61-b19f-ea408b9bda3b
  {
      "Version": 1
  }
+
These two secrets will be consumed in the Pod.

=== Consume secrets in a Pod

The directory `images/parameter-store-kubernetes` contains a Java application that reads secrets from AWS Parameter Store. This application is then packaged as a Pod and deployed in the cluster.

The Pod configuration is shown:

  apiVersion: v1
  kind: Pod
  metadata:
    name: pod-parameter-store
  spec:
    containers:
    - name: pod-parameter-store
      image: arungupta/parameter-store-kubernetes:latest
    restartPolicy: Never

Create the Pod:

  $ kubectl apply -f templates/pod-parameter-store.yaml
  pod "pod-parameter-store" configured

Check the logs of the Pod:

  $ kubectl logs pod-parameter-store
  parameter store: HelloWorld

This shows that the Java application has been able to read both the NAME and GREETING secrets from AWS Parameter Store.

== Secrets using AWS Secrets Manager

In this section, we will create a secret using https://aws.amazon.com/secrets-manager/[AWS Secrets Manager] in the region of choice, and access the secret in a Node.js application deployed within Kubernetes pod. AWS Secrets Manager is available in https://docs.aws.amazon.com/general/latest/gr/rande.html#asm_region[most AWS regions].

AWS Secrets Manager enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. The service integrates with KMS, which uses a https://aws.amazon.com/blogs/security/aws-key-management-service-now-offers-fips-140-2-validated-cryptographic-modules-enabling-easier-adoption-of-the-service-for-regulated-workloads/[FIPS 140-2 validated Hardware Security Module], to provide robust key management controls to secure the secret. AWS Secrets Manager also integrates with AWS IAM and AWS CloudTrail to provide fine-grained access, audit and alerting integration.

=== Update the IAM role for EKS or `kops` Kubernetes Cluster

==== EKS Kubernetes Cluster
EC2 worker nodes use `NodeInstanceRole` created in Step 3 of the https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html[EKS Getting Started guide]. This role must be updated to allow the worked nodes to read the secrets from Secrets Manager.

In the IAM Console, click `roles` and type `NodeInstanceRole` and click it. In the Permissions tab, expand the inline policy and click `Edit policy`. Add the `secretsManager:GetSecretValue` permission to the policy so the policy looks similar to the one below.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "secretsmanager:GetSecretValue",
                    "secretsmanager:DescribeSecret"
                ],
                "Resource": [
                    "arn:aws:secretsmanager:us-west-2:<account-id>:secret:<secret-name>"
                ]
            }
        ]
    }

==== `kops` Kubernetes Cluster

EC2 worker nodes use an instance profile to allow the EC2 node instances to access other AWS services. This role must be updated to allow the worker nodes to read the secrets from Secrets Manager.

In the IAM Console click `roles` and type `nodes` into the search box. Find the `nodes.example.cluster.k8s.local` role
and click it. In the `Permissions` tab, expand the inline policy for `nodes.example.cluster.k8s.local` and click
`Edit policy`.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "secretsmanager:GetSecretValue",
                    "secretsmanager:DescribeSecret"
                ],
                "Resource": [
                    "arn:aws:secretsmanager:us-west-2:<account-id>:secret:<secret-name>"
                ]
            }
        ]
    }

=== Create secrets

. Create a secret key-value pair using https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html[AWS Secrets Manager CLI]. Replace `<SECRETNAME>` and `<REGION>` with your preference.

  aws secretsmanager create-secret --name <SECRETNAME> --description "EKS/kops Demo Secret" --secret-string [{"testkey1":"testvalue1"},{"testkey2":"testvalue2"}] --region <REGION>

. Get the value of created secret using https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/get-secret-value.html[GetSecretValue] API call.

  aws secretsmanager get-secret-value --secret-id <SECRETNAME> --region <REGION>

. For the selected `<REGION>`, AWS Secrets Manager `<ENDPOINT>` can be determined from https://docs.aws.amazon.com/general/latest/gr/rande.html#asm_region[AWS Documentation].

. Note the `ENDPOINT`, `REGION` and `SECRETNAME` values. They will be passed as environment variables in a `.yaml` file described in the next section.

=== Consume secrets in a Pod

The Github repository directory `images/sec_mgr_app` contains a Node.js sample application that reads a secret from AWS Secrets Manager from specified region. This application is then packaged as a Pod and deployed in the cluster.

The Pod configuration is shown below. The `ENDPOINT`, `REGION` and `SECRETNAME` variables are passed as environment variables to the docker image. Change the values of these environment variables to match the values used during creation of secret in AWS Secrets Manager.

    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-secretsmanager
    spec:
      containers:
      - name: pod-secretsmanager
        image: paavanmistry/node-aws-sm-demo:latest
        env:
          - name: ENDPOINT
            value: "https://secretsmanager.us-west-2.amazonaws.com"
          - name: REGION
            value: "us-west-2"
          - name: SECRETNAME
            value: "sm-demo-secret"
      restartPolicy: Never

Create the Pod:

  $ kubectl apply -f templates/pod-secretsmanager.yaml
  pod "pod-secretsmanager" configured

Check the logs of the Pod:

  $ kubectl logs pod-secretsmanager
  Secret retrieved from AWS SecretsManager: The Secret <SECRETNAME> is [{testkey1}:{testvalue1},{testkey2:testvalue2}]

Clean up:

  - `$ kubectl delete -f templates/pod-secretsmanager.yaml`
  - `$ aws secretsmanager delete-secret --secret-id $SECRETNAME --region $REGION`
  - Delete IAM role policy updates for AWS Secrets Manager

== Secrets using Vault

https://www.vaultproject.io/[Hashicorp Vault] is a tool for managing secrets. It secures, stores and tightly controls access to tokens, passwords, certificates, API keys and other secrets.

This section explains how to install and configure Vault on AWS, store secrets, and access them in a Pod. The instructions are inspired from https://github.com/briankassouf/vault-kubernetes-demo.

=== Create EC2 instance

We need an EC2 instance for hosting Vault server. This server needs to be accessible to Kubernetes cluster.

. Create an EC2 instance with Linux flavor. For example `m4.large` with `Amazon Linux`
.. Make sure to allow port `8200` as part of `Configure Security Group`
.. Configure security group to allow 8200 (not TLS by default, more config required for TLS)
.. SSH into the machine:
+
```
ssh -i ~/.ssh/arun-us-east1.pem ec2-user@ec2-54-237-223-40.compute-1.amazonaws.com
```
+
. Note down the private IP address of the EC2 console. This is needed to start our Vault server.

=== Start Vault Server on EC2

. Download Vault server:

  wget https://releases.hashicorp.com/vault/0.9.0/vault_0.9.0_linux_amd64.zip

. Unzip Vault: `unzip vault_0.9.0_linux_amd64.zip`
. Start Vault server:
+
```
[ec2-user@ip-172-31-26-180 ~]$ ./vault server -dev-listen-address=ip-172-31-26-180.ec2.internal:8200 -dev &
[1] 26687
[ec2-user@ip-172-31-26-180 ~]$ ==> Vault server configuration:

                     Cgo: disabled
         Cluster Address: https://ip-172-31-26-180.ec2.internal:8201
              Listener 1: tcp (addr: "ip-172-31-26-180.ec2.internal:8200", cluster address: "172.31.26.180:8201", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
        Redirect Address: http://ip-172-31-26-180.ec2.internal:8200
                 Storage: inmem
                 Version: Vault v0.9.0
             Version Sha: bdac1854478538052ba5b7ec9a9ec688d35a3335

==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed.
Vault is configured to only have a single unseal key. The root
token has already been authenticated with the CLI, so you can
immediately begin using the Vault CLI.

The only step you need to take is to set the following
environment variables:

    export VAULT_ADDR='http://ip-172-31-26-180.ec2.internal:8200'

The unseal key and root token are reproduced below in case you
want to seal/unseal the Vault or play with authentication.

Unseal Key: ZBfexpmasu0r4iba+t8tTlm4L5FQJ+JagglEhbfpxkU=
Root Token: 4e93b3c6-c459-f166-e7e9-6c48044cfdb6

==> Vault server started! Log data will stream in below:

2017/11/20 03:34:06.457231 [INFO ] core: security barrier not initialized
2017/11/20 03:34:06.457349 [INFO ] core: security barrier initialized: shares=1 threshold=1
2017/11/20 03:34:06.457475 [INFO ] core: post-unseal setup starting
2017/11/20 03:34:06.470532 [INFO ] core: loaded wrapping token key
2017/11/20 03:34:06.470542 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2017/11/20 03:34:06.471226 [INFO ] core: successfully mounted backend: type=kv path=secret/
2017/11/20 03:34:06.471239 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2017/11/20 03:34:06.471348 [INFO ] core: successfully mounted backend: type=system path=sys/
2017/11/20 03:34:06.471530 [INFO ] core: successfully mounted backend: type=identity path=identity/
2017/11/20 03:34:06.475065 [INFO ] expiration: restoring leases
2017/11/20 03:34:06.475241 [INFO ] rollback: starting rollback manager
2017/11/20 03:34:06.475583 [INFO ] expiration: lease restore complete
2017/11/20 03:34:06.475583 [INFO ] identity: entities restored
2017/11/20 03:34:06.475628 [INFO ] identity: groups restored
2017/11/20 03:34:06.475641 [INFO ] core: post-unseal setup complete
2017/11/20 03:34:06.475778 [INFO ] core: root token generated
2017/11/20 03:34:06.475782 [INFO ] core: pre-seal teardown starting
2017/11/20 03:34:06.475783 [INFO ] core: cluster listeners not running
2017/11/20 03:34:06.475790 [INFO ] rollback: stopping rollback manager
2017/11/20 03:34:06.475848 [INFO ] core: pre-seal teardown complete
2017/11/20 03:34:06.475905 [INFO ] core: vault is unsealed
2017/11/20 03:34:06.475919 [INFO ] core: post-unseal setup starting
2017/11/20 03:34:06.475965 [INFO ] core: loaded wrapping token key
2017/11/20 03:34:06.475967 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2017/11/20 03:34:06.476108 [INFO ] core: successfully mounted backend: type=kv path=secret/
2017/11/20 03:34:06.476186 [INFO ] core: successfully mounted backend: type=system path=sys/
2017/11/20 03:34:06.476318 [INFO ] core: successfully mounted backend: type=identity path=identity/
2017/11/20 03:34:06.476328 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2017/11/20 03:34:06.476889 [INFO ] expiration: restoring leases
2017/11/20 03:34:06.476945 [INFO ] rollback: starting rollback manager
2017/11/20 03:34:06.477008 [INFO ] identity: entities restored
2017/11/20 03:34:06.477015 [INFO ] identity: groups restored
2017/11/20 03:34:06.477022 [INFO ] core: post-unseal setup complete
2017/11/20 03:34:06.477105 [INFO ] expiration: lease restore complete
```
+
. Run the command to configure Vault CLI to identify the server:
+
```
export VAULT_ADDR='http://ip-172-31-26-180.ec2.internal:8200'
```
+
. Check status:
+
```
[ec2-user@ip-172-31-26-180 ~]$ ./vault status
Type: shamir
Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
Unseal Nonce:
Version: 0.9.0
Cluster Name: vault-cluster-01043c83
Cluster ID: 89ccbeb4-8af1-7dca-77bb-38f39c423a39

High-Availability Enabled: false
```

=== Configure Vault CLI on your local machine

. Download Vault server:

  wget https://releases.hashicorp.com/vault/0.9.0/vault_0.9.0_linux_amd64.zip

. Unzip Vault: `unzip vault_0.9.0_linux_amd64.zip`

. Find public IP address of the EC2 instance and setup an environment variable:

  export VAULT_ADDR='<public-ip-address>'
+
For example, this command will look like:
+
  export VAULT_ADDR='http://http://ec2-54-237-223-40.compute-1.amazonaws.com:8200'
+
. Verify the status can be seen from your local machine:
+
```
$ vault status
Type: shamir
Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
Unseal Nonce:
Version: 0.9.0
Cluster Name: vault-cluster-01043c83
Cluster ID: 89ccbeb4-8af1-7dca-77bb-38f39c423a39

High-Availability Enabled: false
```

. Authenticate against Vault using the root token from the output when starting vault:
+
```
vault auth
Token (will be hidden):
Successfully authenticated! You are now logged in.
token: 4e93b3c6-c459-f166-e7e9-6c48044cfdb6
token_duration: 0
token_policies: [root]
```

=== Configure Kubernetes Service Account

. Create the service account to verify service account token during login:

  $ kubectl create -f templates/vault-reviewer.yaml
  serviceaccount "vault-reviewer" created

. Create the RBAC role that will be used by the service account to access the TokenReview API:

  $ kubectl apply -f templates/vault-reviewer-rbac.yaml
  clusterrolebinding "role-tokenreview-binding" created

. Creat a service account that will be used to login to the auth backend:

  $ kubectl create -f templates/vault-auth.yaml
  serviceaccount "vault-auth" created

=== Configure Kubernetes Auth backend

Service account token, Kubernetes API server address and the certificate used to access the API server are needed in order to configure the Kubernetes Auth backend. Let's get these values.

. On the local machine, read the service account token:

  kubectl get secret \
  $(kubectl get serviceaccount vault-reviewer -o jsonpath={.secrets[0].name}) \
  -o jsonpath={.data.token} | base64 -D -
  export REVIEWER_TOKEN=$(kubectl get secret \
  $(kubectl get serviceaccount vault-reviewer \
  -o jsonpath={.secrets[0].name}) -o jsonpath={.data.token} | base64 -D -)
  && echo $REVIEWER_TOKEN
  eyJ . . . reg

. Get the API server address:

  $ kubectl config view -o jsonpath='{.clusters[*].cluster.server}'
  https://api-example-cluster-k8s-l-1dt7vk-41321592.us-east-1.elb.amazonaws.com https://192.168.99.100:8443
+
This is the address of API servers currently configured. The first one is for the cluster created by kops. Second one is for the minikube server, if its running. The first one is relevant for our case.
+
. Extract the certificate
.. Find the default secret token:

  $ kubectl get secrets | grep default
  default-token-kvjn9          kubernetes.io/service-account-token   3         4d

.. Use the default token name to extract the certificate:

  $ kubectl get secrets default-token-kvjn9 -o jsonpath="{.data['ca\.crt']}" | base64 -D > ~/.kube/kops.crt

. Now that all the required values are available, configure the Kubernetes auth backend.
.. Mount the Kubernetes auth backend:

  $ vault auth-enable kubernetes
  Successfully enabled 'kubernetes' at 'kubernetes'!

.. Configure the auth backend:

  $ vault write auth/kubernetes/config \
    token_reviewer_jwt=$REVIEWER_TOKEN  \
    kubernetes_host=<api-server> \
    kubernetes_ca_cert=@~/.kube/kops.crt
+
For example, here is how our command will look like:

  $ vault write auth/kubernetes/config \
    token_reviewer_jwt=eyJ . . . reg  \
    kubernetes_host=https://api-example-cluster-k8s-l-1dt7vk-41321592.us-east-1.elb.amazonaws.com \
    kubernetes_ca_cert=@~/.kube/kops.crt
  Success! Data written to: auth/kubernetes/config

. Create a role with service account name `vault-auth` in the `default` namespace:

  $ vault write auth/kubernetes/role/demo \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces=default \
    policies=kube-auth \
    period=60s
  Success! Data written to: auth/kubernetes/role/demo

. Read the role:

  $ vault read auth/kubernetes/role/demo
  Key                               Value
  ---                               -----
  bound_service_account_names       [vault-auth]
  bound_service_account_namespaces  [default]
  max_ttl                           0
  num_uses                          0
  period                            60
  policies                          [kube-auth]
  ttl                               0

. Create a policy for this role

  $ vault policy-write kube-auth templates/kube-auth.hcl
  Policy 'kube-auth' written.

. Write secrets to Vault:

  $ vault write secret/creds GREETING=Hello NAME=World
  Success! Data written to: secret/creds

. Check that this value can be read:

  $ vault read -field=GREETING secret/creds
  Hello

=== Deploy a Pod using secrets from Vault

Let's deploy a Pod that is reading secrets from the Vault server. Here is the sequence of events that need to happen:

- Pod needs to know the address of Vault server. This is passed as `VAULT_ADDR` environment variable.
- Pod reads the Kubernetes service account token
- Service account token is passed to Vault server to retrieve a client token
- Client token is used to authenticate and read secrets from Vault

More details about the Docker image used in the Pod is at https://github.com/arun-gupta/vault-kubernetes.

. The Pod configuration file looks like:

  apiVersion: v1
  kind: Pod
  metadata:
    name: vault-kubernetes
  spec:
    containers:
    - name: vault-kubernetes
      image: arungupta/vault-kubernetes:latest
      env:
        - name: VAULT_ADDR
          valueFrom:
            configMapKeyRef:
              name: vault
              key: address
    restartPolicy: Never

. Create the ConfigMap:

  $ kubectl create configmap vault --from-literal=address=$VAULT_ADDR
  configmap "vault" created

. Deploy the Pod:

  $ kubectl apply -f templates/pod-vault.yaml
  pod "vault-kubernetes" created

. Get the list of Pods:

  $ kubectl get pods --show-all
  NAME               READY     STATUS      RESTARTS   AGE
  vault-kubernetes   0/1       Completed   0          20s

. Get logs from the completed Pod:

  $ kubectl logs vault-kubernetes
  Connecting to Vault: http://ec2-54-237-223-40.compute-1.amazonaws.com:8200
  vault: HelloWorld


You are now ready to continue on with the workshop!

:frame: none
:grid: none
:valign: top

[align="center", cols="2", grid="none", frame="none"]
|=====
|image:button-continue-developer.png[link=../../04-path-security-and-networking/402-authentication-and-authorization/]
|image:button-continue-operations.png[link=../../04-path-security-and-networking/402-authentication-and-authorization/]
|link:../../developer-path.adoc[Go to Developer Index]
|link:../../operations-path.adoc[Go to Operations Index]
|=====
