= Admission Control for Kubernetes on AWS
:toc:
:icons:
:linkcss:
:imagesdir: ../../resources/images

Doc Writer <tim@styra.com>
v1.0, 2018-01-17

Kubernetes https://kubernetes.io/docs/admin/admission-controllers/[Admission Controllers]
perform *semantic validation* of resources during create, update, and delete operations.
In Kubernetes 1.8+, you can use the http://www.openpolicyagent.org/[Open Policy Agent] and
https://kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks[Kubernetes External Admission Webhooks]
to enforce custom admission control policies without recompiling or reconfiguring
the Kubernetes API server.

The Open Policy Agent is a general-purpose policy engine that aims to policy-enable
other projects or services.  It provides a rich, declarative policy language
where policy decisions may be booleans (e.g. allow/deny), strings (e.g. hostnames),
numbers (e.g. ratelimits), arrays (e.g. destination hosts for a pod), or dictionaries
(e.g. JSON patch changes to a pod definition).

For example, using OPA with Kubernetes you could enforce any of the following policies:

* All images come from an AWS repository (other than a whitelist)
* Images are pulled from the same AWS repository in the same region as the cluster
* Each namespace is controlled by the users in different AWS IAM groups


== Goals

This tutorial shows how to use OPA to enforce custom policies on resources in
Kubernetes running on AWS. For the purpose of this tutorial, you will define a policy that
requires all pod images to come from an AWS image repository.


== Prerequisites

This tutorial requires a Kubernetes 1.8 cluster.
Keep in mind that External Admission Webhook support in Kubernetes is currently in **alpha**.

== Overview

. Create the Kubernetes cluster with `kops` and enable the External Admission Webhooks
. Install the OpenPolicyAgent as an External Admission Webhook
. Write admission control policies with the OpenPolicyAgent

== Step 1: Create Cluster with External Admission Controllers via `kops`

Create a Kubernetes 1.8 cluster with `kops` as shown in the
link:../cluster-install/readme.adoc[Cluster Install] instructions.

Update the Cluster spec to enable webhooks. (Remember that when you set up
  the cluster you chose the name `example.cluster.k8s.local`. )

```bash
kops edit example.cluster.k8s.local
```

```yaml
  kubeAPIServer:
    admissionControl:
      - NamespaceLifecycle
      - LimitRanger
      - ServiceAccount
      - PersistentVolumeLabel
      - DefaultStorageClass
      - DefaultTolerationSeconds
      - GenericAdmissionWebhook
      - ResourceQuota
    runtimeConfig:
      admissionregistration.k8s.io/v1alpha1: "true"
```

Then update `kops`.
```bash
kops update example.cluster.k8s.local --yes
```


== Step 2: Deploy OPA on Kubernetes

=== 2.1 Generate service-certificate for OPA

First, create the required OpenSSL configuration files.  Put these in your current
directory; we will only use them once to generate certificates.  You may discard
them after running the `openssl` commands below.

*server.conf*
```bash
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
```

Now create local files that contain the CA and server key pair.

Create a certificate authority:

```bash
openssl genrsa -out ca.key 2048
```

```bash
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
```

Create a server certiticate.
IMPORTANT: the CN must match the name of the service used to expose the external admission controller.

```bash
openssl genrsa -out server.key 2048
```

```bash
openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
```

```bash
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf
```

=== 2.2: Deploy OPA on Kubernetes

First, create a namespace to deploy OPA into.

```bash
kubectl create namespace opa
```

Create a Service to expose the OPA API. The Kubernetes API server will lookup
the Service and execute webhook requests against it.

**opa-admission-controller-service.yaml**:

```yaml
kind: Service
apiVersion: v1
metadata:
  name: opa
spec:
  selector:
    app: opa
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
```

```bash
kubectl create -f opa-admission-controller-service.yaml -n opa
```

Next, create Secrets containing the TLS credentials for OPA:

```bash
kubectl create secret generic opa-ca --from-file=ca.crt -n opa
kubectl create secret tls opa-server --cert=server.crt --key=server.key -n opa
```

Finally, create the Deployment to run OPA as an Admission Controller.
The deployment contains two containers: `opa` and `kube-mgmt`.  `opa` by
itself is a general-purpose policy engine and knows nothing about Kubernetes.
`kube-mgmt` is a collection of Kubernetes-specific code that helps
OPA interact with kubernetes.

**opa-admission-controller-deployment.yaml**:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  name: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        - name: opa
          image: openpolicyagent/opa:0.5.13
          args:
            - "run"
            - "--server"
            - "--tls-cert-file=/certs/tls.crt"
            - "--tls-private-key-file=/certs/tls.key"
            - "--addr=0.0.0.0:443"
            - "--insecure-addr=127.0.0.1:8181"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-server
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:0.5
          args:
            - "--replicate=v1/pods"
            - "--pod-name=$(MY_POD_NAME)"
            - "--pod-namespace=$(MY_POD_NAMESPACE)"
            - "--register-admission-controller"
            - "--admission-controller-ca-cert-file=/certs/ca.crt"
            - "--admission-controller-service-name=opa"
            - "--admission-controller-service-namespace=$(MY_POD_NAMESPACE)"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-ca
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      volumes:
        - name: opa-server
          secret:
            secretName: opa-server
        - name: opa-ca
          secret:
            secretName: opa-ca
```

```bash
kubectl create -f opa-admission-controller-deployment.yaml -n opa
```

When OPA starts, the sidecar (`kube-mgmt`) will register it as an External
Admission Controller. To verify that registration succeeded, query
 the Kubernetes API for the list of External Admission Controllers.

```bash
kubectl describe externaladmissionhookconfigurations admission.openpolicyagent.org
```



Finally, you can follow the OPA logs to see the webhook requests being issued
by the Kubernetes API server:

```
kubectl logs -l app=opa -c opa -n opa
```

== Step 3:  Enforce Kubernetes Admission Control with OPA

=== 3.1 Load a policy into OPA
To test admission control, create a policy that requires all images
to come from an AWS repository.  For details on the policy language, see the
http://www.openpolicyagent.org/docs/[Open Policy Agent] documentation.

NOTE: Below replace the Amazon account ID 123456789 with your own account if your
want the pod to actually come up.  If you just want to see the admission
controller in action, you can leave it with the fake ID.  That account ID
also appears in the image names when you create pods below; just make sure
the account IDs are the same.

**image_source.rego**:

```ruby
package system

# Deny requests that include container images not from ECR.
deny[explanation] {
    image_name = input.spec.object.Spec.Containers[_].Image
    image_name_parts = split(image_name, "/")
    repo_name = image_name_parts[0]
    not startswith(repo_name, "12345678.dkr.ecr.us-west-2.amazonaws.com")
    explanation = sprintf("image '%v' not from AWS ECR", [image_name])
}


# main is entry point to policy.
# Boilerplate required by admission webhook.
# Actual policy decision is `status`, which takes the form
#   {"allowed": BOOLEAN, "status": {"reason": STRING}}
main = {
    "apiVersion": "admission.k8s.io/v1alpha1",
    "kind": "AdmissionReview",
    "status": {"allowed": allowed, "status": {"reason": reason}}
}

# Boilerplate: construct 'reason' and 'allowed' variables.
#  Real policy is the collection of 'deny' statements above.
#  If not denied, allow.
reason = msg {
    msg = concat(", ", deny)
}
default allowed = true
allowed = false { n = count(deny); n > 0 }

```

Store the policy in Kubernetes as a ConfigMap.

```bash
kubectl create configmap image-source --from-file=image_source.rego -n opa
```

The OPA sidecar will notice the ConfigMap and automatically load the contained
policy into OPA.

=== 3.2 Check that the policy is working

To verify that your policy is working, create separate test pods.

**nginx-pod.yaml**:

```yaml
kind: Pod
apiVersion: v1
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - image: nginx
    name: nginx
```

NOTE: Below replace the Amazon account ID 123456789 with your own account if your
want the pod to actually come up.  If you just want to see the admission
controller in action, you can leave it with the fake ID.

**amazon-linux-pod.yaml**:

```yaml
kind: Pod
apiVersion: v1
metadata:
  name: amazon-linux-pod
  labels:
    app: amazon-linux
spec:
  containers:
  - image: 123456789.dkr.ecr.us-west-2.amazonaws.com/amazon-linux
    name: amazon-linux
```

Verify that you can create an amazon-linux pod.
```bash
kubectl create -f amazon-linux-pod.yaml
```

Verify that you CANNOT create an nginx pod and receive the appropriate error message.
```bash
kubectl create -f nginx-pod.yaml
```
```bash
Error from server (image 'nginx' not from AWS ECR): error when creating "nginx-pod.yaml":
```

This example shows how to ensure ALL images come from an AWS repository.
But in reality you might have a collection of images like `nginx` that
can come from outside of AWS.  Or maybe you only want to apply the policy
to certain Kubernetes namespaces.  OPA's policy language is flexible enough
to add image whitelists and control the applicable namespaces.
Just modify your policy locally, update the ConfigMap, and `kube-mgmt` will
update OPA with your changes.

== Wrap Up

Congratulations for finishing the tutorial!

This tutorial showed how you can leverage OPA to enforce admission control
decisions in Kubernetes clusters without modifying or recompiling any
Kubernetes components. Furthermore, once Kubernetes is configured to use OPA as
an External Admission Controller, policies can be modified on-the-fly to
satisfy changing operational requirements.

You are now ready to continue on with the workshop!

:frame: none
:grid: none
:valign: top

[align="center", cols="2", grid="none", frame="none"]
|=====
|image:button-continue-developer.png[link=../../04-path-security-and-networking/404-network-policies/]
|image:button-continue-operations.png[link=../../04-path-security-and-networking/404-network-policies/]
|link:../../developer-path.adoc[Go to Developer Index]
|link:../../operations-path.adoc[Go to Operations Index]
|=====