= Service Mesh integration with Kubernetes
:toc:
:icons:
:linkcss:
:imagesdir: ../../resources/images

In this section, we will demonstrate how to integrate service mesh components with a Kubernetes cluster.
Service mesh is a layer that manages communication between microservices and it is becoming
increasingly popular for cloud native applications. Some of the critical features are service discovery,
load balancing, automatic retries, circuit breakers, collect request/response metrics and tracing
info.

We will look into two well-known service mesh integrations:

. Linkerd
. Istio

== Using Linkerd with Kubernetes

https://linkerd.io/[Linkerd] is a layer 5/7 proxy that routes and load balances traffic over HTTP, Thrift, Mux, HTTP/2 and gRPC. Its based on Finagle (built by Twitter), and in January 2017 linkerd became a member of the link:https://www.cncf.io/[CNCF], alongside Kubernetes.

https://linkerd.io/[Linkerd] adds visibility, control, and reliability to your application with
a wide array of powerful techniques: circuit-breaking, latency-aware load balancing, eventually
consistent (“advisory”) service discovery, deadline propagation, and tracing and instrumentation.

In this exercise, we will focus on visibility for your services running in a Kubernetes cluster. We will
install linkerd in your k8s cluster, run a couple of simple microservices and demonstrate how
linkerd captures top-line service metrics such as success rates, request volumes and latencies.

In this exercise we'll look at a few of the features linkerd provides, such as:

. Monitoring the traffic within the service mesh
. Per request routing

=== Create a Kubernetes cluster

In order to perform exercises in this chapter, you’ll need to deploy configurations to a Kubernetes cluster. To create an EKS-based Kubernetes cluster, use the link:../../01-path-basics/102-your-first-cluster#create-a-kubernetes-cluster-with-eks[AWS CLI] (recommended).  If you wish to create a Kubernetes cluster without EKS, you can instead use link:../../01-path-basics/102-your-first-cluster#alternative-create-a-kubernetes-cluster-with-kops[kops].

=== Install linkerd

Install linkerd using `kubectl`. This will install linkerd as a DaemonSet (i.e., one instance per
host) running in the default Kubernetes namespace.

  kubectl apply -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
  k8s-daemonset/k8s/linkerd.yml
  configmap "l5d-config" created
  daemonset "l5d" created
  service "l5d" created

Check that the linkerd (named as `l5d`) pods are running:

  $ kubectl get pods
  NAME        READY     STATUS    RESTARTS   AGE
  l5d-4kg47   2/2       Running   0          55s
  l5d-d0wvv   2/2       Running   0          55s
  l5d-gfpc7   2/2       Running   0          55s
  l5d-w0d95   2/2       Running   0          55s
  l5d-xppqs   2/2       Running   0          55s

This is the output shown from a 5 worker node cluster.

Check that the linkerd service is running:

  $ kubectl get svc
  NAME         TYPE           CLUSTER-IP     EXTERNAL-IP        PORT(S)                                        AGE
  kubernetes   ClusterIP      100.64.0.1     <none>             443/TCP                                        15h
  l5d          LoadBalancer   100.68.65.99   a306c8f52b92c...   4140:30609/TCP,4141:31489/TCP,9990:31710/TCP   24s

Get more details about linkerd service:

  $ kubectl describe svc/l5d
  Name:                     l5d
  Namespace:                default
  Labels:                   <none>
  Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"l5d","namespace":"default"},"spec":{"ports":[{"name":"outgoing","port":4140},{...
  Selector:                 app=l5d
  Type:                     LoadBalancer
  IP:                       100.70.56.141
  LoadBalancer Ingress:     abf44db7dbe1211e7a7d00220684043f-1319890531.eu-central-1.elb.amazonaws.com
  Port:                     outgoing  4140/TCP
  TargetPort:               4140/TCP
  NodePort:                 outgoing  30108/TCP
  Endpoints:                100.96.3.2:4140,100.96.4.2:4140,100.96.5.4:4140 + 2 more...
  Port:                     incoming  4141/TCP
  TargetPort:               4141/TCP
  NodePort:                 incoming  31209/TCP
  Endpoints:                100.96.3.2:4141,100.96.4.2:4141,100.96.5.4:4141 + 2 more...
  Port:                     admin  9990/TCP
  TargetPort:               9990/TCP
  NodePort:                 admin  32117/TCP
  Endpoints:                100.96.3.2:9990,100.96.4.2:9990,100.96.5.4:9990 + 2 more...
  Session Affinity:         None
  External Traffic Policy:  Cluster
  Events:
    Type    Reason                Age   From                Message
    ----    ------                ----  ----                -------
    Normal  CreatingLoadBalancer  3m    service-controller  Creating load balancer
    Normal  CreatedLoadBalancer   3m    service-controller  Created load balancer

You can go to linkerd's admin page (i.e ELB:9990) to verify installation. It may take a minute or two before the
ELB DNS is available. If you have an issue accessing the load balancer endpoint, it could be because your firewall
or VPN is preventing access to port 9990.

  LINKERD_ELB=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}")
  open http://$LINKERD_ELB:9990

The default dashboard looks like as shown:

image::linkerd-default-dashboard.png[]

=== Install sample microservices apps

The https://github.com/linkerd/linkerd-examples/tree/master/k8s-daemonset/k8s[Github repo for
linkerd examples] has two microservices apps called "`hello`" and "`world`". They both communicate
with each other to complete the request. Run this command to install these apps in the default
namespace.

  $ kubectl apply -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
  k8s-daemonset/k8s/hello-world.yml
  replicationcontroller "hello" created
  service "hello" created
  replicationcontroller "world-v1" created
  service "world-v1" created

Generate some traffic by running this command:

  http_proxy=$LINKERD_ELB:4140 curl -s http://hello

Linkerd will show the number of requests being served, connections and a bunch of rich data:

image::linkerd.png[]

=== Install linkerd-viz

https://github.com/linkerd/linkerd-viz[linkerd-viz] is a monitoring application based on https://prometheus.io/[Prometheus] and http://grafana.org/[Grafana]. It can automatically find linkerd instances and services
that are installed in your k8s cluster.

  $ kubectl apply -f https://raw.githubusercontent.com/linkerd/linkerd-viz/master/k8s/linkerd-viz.yml
  replicationcontroller "linkerd-viz" created
  service "linkerd-viz" created

You can open linkerd-viz ELB to view the dashboard:

  LINKERD_VIZ_ELB=$(kubectl get svc linkerd-viz -o jsonpath="{.status.loadBalancer.ingress[0].*}")
  open http://$LINKERD_VIZ_ELB

As with the previous example, it may take a minute or two before the ELB DNS is available.

image::linkerd-viz.png[]

=== Per request routing

We'll use the same "`hello-world`" application used in the example above, but this time we'll deploy version 2 of the
"`world`" microservice, and we'll specify on a per request level whether the request should use v1 or v2 of the "`"world`" microservice.

If you haven't already deployed the "`hello-world`" application, deploy it now.

    kubectl apply -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
    k8s-daemonset/k8s/hello-world.yml

Delete the previous linkerd Daemonset, as we're going to update the ConfigMap and install a new one:

    $ kubectl delete ds/l5d

Deploy the linkerd ingress so we can access the application externally.

    $ kubectl apply -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
    k8s-daemonset/k8s/linkerd-ingress.yml
    configmap "l5d-config" configured
    daemonset "l5d" configured
    service "l5d" configured

Now deploy version2 of the "`world`" microservice.

    $ kubectl apply -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
    k8s-daemonset/k8s/world-v2.yml
    replicationcontroller "world-v2" created
    service "world-v2" created

Send a request to v1 of the service. It should reply with 'Hello world'.

    INGRESS_LB=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}")
    curl -H 'Host: www.hello.world' $INGRESS_LB

After a minute or two, it should reply with `Hello world` as shown:

  Hello (100.96.1.12) world (100.96.1.14)

Now send a request to v2 of the service by modifying the header in the request.

    curl -H "Host: www.hello.world" -H "l5d-dtab: /host/world => /srv/world-v2;" $INGRESS_LB

It should reply with 'Hello earth' as shown:

  Hello (100.96.1.11) earth (100.96.2.14)

This demonstrates that v1 and v2 of the `world` service are running in the cluster, and you can specify in the
request header which version of the service to route individual requests to.

That's it!

You can look into linkerd configuration files in https://github.com/linkerd/linkerd-examples/tree/master/k8s-daemonset/k8s[linkerd examples] to learn more.

=== Cleanup

Remove the installed components:

  kubectl delete -f https://raw.githubusercontent.com/linkerd/linkerd-viz/master/k8s/linkerd-viz.yml
  kubectl delete -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
  k8s-daemonset/k8s/world-v2.yml
  kubectl delete -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
  k8s-daemonset/k8s/hello-world.yml
  kubectl delete -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
  k8s-daemonset/k8s/linkerd-ingress.yml
  kubectl delete -f https://raw.githubusercontent.com/linkerd/linkerd-examples/master/\
  k8s-daemonset/k8s/linkerd.yml

== Using Istio with Kubernetes

https://istio.io[Istio] is a layer 4/7 proxy that routes and load balances traffic over HTTP, WebSocket, HTTP/2, gRPC and supports application protocols such as MongoDB and Redis. Istio uses the Envoy proxy to manage all inbound/outbound traffic in the service mesh. Envoy was built by https://www.lyft.com/[Lyft], and in Sept 2017 Envoy became a member of the link:https://www.cncf.io/[CNCF], alongside Kubernetes. Istio is a joint collaboration between Google, IBM and Lyft.

Istio has a wide variety of traffic management features that live outside the application code, such
as A/B testing, phased/canary rollouts, failure recovery, circuit breaker, layer 7 routing and policy enforcement
(all provided by the Envoy proxy). Istio also supports ACLs, rate limits, quotas, authentication, request tracing and
telemetry collection using its Mixer component. The goal of the Istio project is to support traffic management and
security of microservices without requiring any changes to the application; it does this by injecting a sidecar into
your pod that handles all network communications.

In this exercise we'll look at a few of the features Istio provides, such as:

. Weighted routing
. Distributed tracing
. Mutual TLS authentication

=== Create Kubernetes cluster

In order to perform exercises in this chapter, you’ll need to deploy configurations to a Kubernetes cluster. To create an EKS-based Kubernetes cluster, use the link:../../01-path-basics/102-your-first-cluster#create-a-kubernetes-cluster-with-eks[AWS CLI] (recommended).  If you wish to create a Kubernetes cluster without EKS, you can instead use link:../../01-path-basics/102-your-first-cluster#alternative-create-a-kubernetes-cluster-with-kops[kops].

=== Install Istio

Istio requires a binary installed on your laptop in order to inject the Envoy sidecar into your pods. This means
you'll need to download Istio. Istio can also automatically inject the sidecar; for more info see the
https://istio.io/docs/setup/kubernetes/quick-start.html[Istio quick start]

    curl -L https://git.io/getLatestIstio | sh -
    cd istio-*
    export PATH=$PWD/bin:$PATH

You should now be able to run the `istioctl` CLI

  $ istioctl version
  Version: 0.2.10
  GitRevision: f27f2803f59994367c1cca47467c362b1702d605
  GitBranch: release-0.2
  User: sebastienvas@ee792364cfc2
  GolangVersion: go1.8

Install Istio using `kubectl`. This will install Istio into its own namespace, `istio-system`. Change to the
directory where you downloaded Istio in the step above.

    kubectl apply -f install/kubernetes/istio.yaml

Check the Istio has been installed. Note that Istio is installed into its own namespace.

```
$ kubectl get all --namespace istio-system
NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/istio-ca        1         1         1            1           1m
deploy/istio-egress    1         1         1            1           1m
deploy/istio-ingress   1         1         1            1           1m
deploy/istio-mixer     1         1         1            1           2m
deploy/istio-pilot     1         1         1            1           1m

NAME                          DESIRED   CURRENT   READY     AGE
rs/istio-ca-2651333813        1         1         1         1m
rs/istio-egress-2836352731    1         1         1         1m
rs/istio-ingress-2873642151   1         1         1         1m
rs/istio-mixer-1999632368     1         1         1         2m
rs/istio-pilot-1811250569     1         1         1         1m

NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/istio-ca        1         1         1            1           1m
deploy/istio-egress    1         1         1            1           1m
deploy/istio-ingress   1         1         1            1           1m
deploy/istio-mixer     1         1         1            1           2m
deploy/istio-pilot     1         1         1            1           1m

NAME                                READY     STATUS    RESTARTS   AGE
po/istio-ca-2651333813-pcr1f        1/1       Running   0          1m
po/istio-egress-2836352731-sfj7j    1/1       Running   0          1m
po/istio-ingress-2873642151-vzfxr   1/1       Running   0          1m
po/istio-mixer-1999632368-nz0mw     2/2       Running   0          2m
po/istio-pilot-1811250569-mmfdg     1/1       Running   0          1m
```

=== Deploying the sample application

We'll use a sample application developed by the Istio team to check out the Istio features. Since we are using
the manual method of injecting the Envoy sidecar into the application, we need to use the `istioctl` as shown below.

    kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/kube/bookinfo.yaml)

This will deploy the BookInfo application, which consists of 4 microservices, each written using a different language,
which collaborate to show book product information, book details and book reviews. Each microservice is deployed in its own pod, with the Envoy proxy injected into the pod; Envoy will now take over all network communications between
the pods.

Let's check that all components were installed

```
$ kubectl get all
NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/details-v1       1         1         1            1           3h
deploy/productpage-v1   1         1         1            1           3h
deploy/ratings-v1       1         1         1            1           3h
deploy/reviews-v1       1         1         1            1           3h
deploy/reviews-v2       1         1         1            1           3h
deploy/reviews-v3       1         1         1            1           3h

NAME                           DESIRED   CURRENT   READY     AGE
rs/details-v1-39705650         1         1         1         3h
rs/productpage-v1-1382449686   1         1         1         3h
rs/ratings-v1-3906799406       1         1         1         3h
rs/reviews-v1-2953083044       1         1         1         3h
rs/reviews-v2-348355652        1         1         1         3h
rs/reviews-v3-4088116596       1         1         1         3h

NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/details-v1       1         1         1            1           3h
deploy/productpage-v1   1         1         1            1           3h
deploy/ratings-v1       1         1         1            1           3h
deploy/reviews-v1       1         1         1            1           3h
deploy/reviews-v2       1         1         1            1           3h
deploy/reviews-v3       1         1         1            1           3h

NAME                                 READY     STATUS    RESTARTS   AGE
po/details-v1-39705650-vc2x0         2/2       Running   0          3h
po/productpage-v1-1382449686-b7frw   2/2       Running   0          3h
po/ratings-v1-3906799406-11pcn       2/2       Running   0          3h
po/reviews-v1-2953083044-sktvt       2/2       Running   0          3h
po/reviews-v2-348355652-xbbbv        2/2       Running   0          3h
po/reviews-v3-4088116596-pkkjk       2/2       Running   0          3h
```

If all components were installed successfully, you should be able to see the product page. This may take a minute or two, first for the Ingress to be created, and secondly for the Ingress to hook up with the services it exposes. Just keep refreshing the browser until the booking product page appears.

  ISTIO_INGRESS=$(kubectl get ingress gateway -o jsonpath="{.status.loadBalancer.ingress[0].*}")
  open http://$ISTIO_INGRESS/productpage

It looks like as shown:

image::istio-sample-app-product-page.png[]

=== Weighted routing

The sample application is pretty useful. You can see in the `kubectl get all` command above that its deployed
more than one version of the 'reviews' microservice. We're going to use weighted routing to route
50% of the traffic to v3 of the reviews microservice. v3 shows stars for each review, whereas v1 does not.
We'll then query the bookinfo product page a few times and count the number of times a review page appears
containing stars for a review; this will indicate we are being routed to v3 of the reviews page.

    $ kubectl create -f samples/bookinfo/kube/route-rule-all-v1.yaml
    routerule "productpage-default" created
    routerule "reviews-default" created
    routerule "ratings-default" created
    routerule "details-default" created
    $ kubectl replace -f samples/bookinfo/kube/route-rule-reviews-50-v3.yaml
    routerule "reviews-default" replaced

The Envoy proxy does not round robin the routing to different versions of the microservice, so if you access
the product page twice you are unlikely to see one request use v1 of reviews, and a second request use v3. However, over a hundred requests 50% of them should be routed to v3 of the reviews page. We can test this using
the script below. Make sure you don't have a file called `mfile` in your current folder before running this.
The script sends 100 `curl` requests to the bookinfo product page, which may take around 30s, and then counts
those which have stars in the response. For the eagle eyed amongst you, the divde by 2 is because the
productpage html contains two reviewers, and we simply want to count the number of `curls` that returned
"`full stars`" in the review page. Out of 100 curls we expect 50 of them to contain "`full stars`".

    ISTIO_INGRESS=$(kubectl get ingress gateway -o jsonpath="{.status.loadBalancer.ingress[0].*}")
    for((i=1;i<=100;i+=1));do curl  -s http://$ISTIO_INGRESS/productpage >> mfile; done;
    a=$(grep 'full stars' mfile | wc -l) && echo Number of calls to v3 of reviews service "$(($a / 2))"

It shows the output as:

    Number of calls to v3 of reviews service 50

Finally, remove the temporary file:

    rm mfile

This weighted routing was handled by Istio routing the traffic between the versions and scaling the reviews
microservice to accommodate the traffic load.

=== Distributed tracing

Istio is deployed as a sidecar proxy into each of your pods; this means it can see and monitor all the traffic flows
between your microservices and generate a graphical representation of your mesh traffic. We'll use the bookinfo
application you deployed in the previous step to demonstrate this.

First, install Prometheus, which will obtain the metrics we need from Istio

    $ kubectl apply -f install/kubernetes/addons/prometheus.yaml
    configmap "prometheus" created
    service "prometheus" created
    deployment "prometheus" created

Check that Prometheus is running:

  $ kubectl -n istio-system get svc prometheus
  NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
  prometheus   ClusterIP   100.69.199.148   <none>        9090/TCP   47s

Now install the Servicegraph addon; Servicegraph queries Prometheus, which obtains details of the mesh traffic flows
from Istio

    $ kubectl apply -f install/kubernetes/addons/servicegraph.yaml
    deployment "servicegraph" created
    service "servicegraph" created

Check that the Servicegraph was deployed:

  $ kubectl -n istio-system get svc servicegraph
  NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
  servicegraph   ClusterIP   100.65.77.1   <none>        8088/TCP   5m

Generate some traffic to the bookinfo application:

  ISTIO_INGRESS=$(kubectl get ingress gateway -o jsonpath="{.status.loadBalancer.ingress[0].*}")
  open http://$ISTIO_INGRESS/productpage

View the Servicegraph UI - we'll use port forwarding to access this

    kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=servicegraph -o jsonpath='{.items[0].metadata.name}') 8088:8088 &
    open http://localhost:8088/dotviz

You should see a distributed trace that looks something like this. It may take a few seconds for Servicegraph to become available, so refresh the browser if you do not receive a response.

image::istio-trace.png[]

=== Enable mutual TLS authentication between sidecars

https://istio.io/docs/concepts/security/mutual-tls.html[Istio-auth] enables secure communication between microservices
by enforcing mutual TLS communication between the sidecar proxies. Implementing this is simple; we simply install
Istio with mutual TLS enabled.

If you have run the examples above, uninstall Istio:

    kubectl delete -f install/kubernetes/istio.yaml

and reinstall it with the Auth module enabled

    kubectl apply -f install/kubernetes/istio-auth.yaml

all traffic between microservices will now be encrypted.

=== Cleanup

Remove the installed components

  kubectl delete -f install/kubernetes/addons/servicegraph.yaml
  kubectl delete -f install/kubernetes/addons/prometheus.yaml
  kubectl delete -f install/kubernetes/istio-auth.yaml
  kubectl delete -f install/kubernetes/istio.yaml
  ./samples/bookinfo/kube/cleanup.sh

Accept the `default` namespace in the cleanup script above.

Some errors may appear in the output when deleting Istio. These are related to Istio components you have not installed, so no need to worry about these. You can confirm that everything has been uninstalled as follows. No Istio or Bookinfo components should remain:

    kubectl get all
    kubectl get all --namespace istio-system


You are now ready to continue on with the workshop!

:frame: none
:grid: none
:valign: top

[align="center", cols="2", grid="none", frame="none"]
|=====
|image:button-continue-standard.png[link=../../02-path-working-with-clusters/203-cluster-upgrades]
|image:button-continue-operations.png[link=../../02-path-working-with-clusters/203-cluster-upgrades]
|link:../../standard-path.adoc[Go to Standard Index]
|link:../../operations-path.adoc[Go to Operations Index]
|=====
