= Using AWS X-Ray to analyze and debug applications
:toc:

This chapter will explain how to use https://aws.amazon.com/xray/[AWS X-Ray] to perform distribute tracing on the applications deployed on Kubernetes. AWS X-Ray helps developers analyze and debug production, distributed applications, such as those built using a microservices architecture. With X-Ray, you can understand how your application and its underlying services are performing to identify and troubleshoot the root cause of performance issues and errors. We will also be using https://aws.amazon.com/ecr/[Amazon EC2 Container Registry] to store our Docker container images. ECR is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images.

== Pre-requisites

In order to perform exercises in this chapter, you’ll need to deploy configurations to a Kubernetes cluster. To create an EKS-based Kubernetes cluster, use the link:/01-path-basics/102-your-first-cluster#create-a-kubernetes-cluster-with-eks[AWS CLI] (recommended). If you wish to create a Kubernetes cluster without EKS, you can instead use link:/01-path-basics/102-your-first-cluster#alternative-create-a-kubernetes-cluster-with-kops[kops].
You will need AWS CLI and Docker installed on your desktop.

The AWS Region where you deployed Kubernetes must have the following services:

* AWS X-Ray
* Amazon EC2 Container Registry

All application code and configuration files for this chapter are in the `x-ray-ecr` directory.

== IAM Roles for Kubernetes worker nodes
We need the following `AWS managed policy` for the worker nodes:

* **AWSXrayWriteOnlyAccess**  (To allow AWS X-Ray daemon running inside Kubernetes worker nodes send traces to AWS X-ray)
* **AmazonEC2ContainerRegistryReadOnly** (To pull Docker images from ECR)

Navigate to the EC2 Console, select any of the worker nodes, under the `Description` tab, click on the link to `IAM role`.

image::images/ec2-iamrole.png[ec2-iamrole, 800]

This will bring you to the IAM console. 
Under the `Permissions` tab, click on `Attach policy` and add the following 2 policies:

* AWSXrayWriteOnlyAccess  
* AmazonEC2ContainerRegistryReadOnly 

image::images/iamrole.png[iamrole, 800]


== Build the AWS X-Ray Daemon

.  Login to Amazon ECR using AWS CLI.

	$ `aws ecr get-login --no-include-email`

.  Create a private Docker repository for our X-Ray daemon

	$ aws ecr create-repository --repository-name xraydaemon

	{
		"repository": {
			"repositoryArn": "arn:aws:ecr:ap-southeast-1:<account-id>:repository/xraydaemon",
			"registryId": "<account-id>",
			"repositoryName": "xraydaemon",
			"repositoryUri": "<account-id>.dkr.ecr.ap-southeast-1.amazonaws.com/xraydaemon",
			"createdAt": 1510724710.0
		}
	}

The **repositoryUri** is the URI of our newly created Docker image repository, it has the format of *_<AWS::AccountId>.dkr.ecr.<AWS::Region>.amazonaws.com/<image name>_*.

.  Build the AWS X-ray daemon docker image locally on your laptop. Replace **repositoryUri** with the value of the output from the last command.

	$ cd x-ray-daemon && docker build -t <repositoryUri>:v1 .

.  Push the xraydaemon docker image to ECR: 

	$ docker push <repositoryUri>:v1

=== Edit templates/daemonsetxray.yaml

Replace line 14 and change *image: <AWS::AccountId>.dkr.ecr.<AWS::Region>.amazonaws.com/xraydaemon:v1* to your xraydaemon repositoryUri.

. Deploy AWS X-Ray as DaemonSet

	$ cd .. && kubectl create -f templates/daemonsetxray.yaml


== Build a Simple Python Flask Application

.  Create a ECR repository for the Python Flask application: 

	$ aws ecr create-repository --repository-name flaskxray

.  Build the Docker image for Python Flask application: 

	$ cd python-flask-app && docker build -t <AWS::AccountId>.dkr.ecr.<AWS::Region>.amazonaws.com/flaskxray:v1 .

.  Test the docker image locally, you should get a reply when you `curl http://0.0.0.0:5000` : 

	$ docker run -p 5000:5000 -it <AWS::AccountId>.dkr.ecr.<AWS::Region>.amazonaws.com/flaskxray:v1

 		Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

.  Push the docker image to ECR: 

	$ docker push <AWS::AccountId>.dkr.ecr.<AWS::Region>.amazonaws.com/flaskxray:latest

=== Edit templates/flaskyxray-deployment.yaml
Replace line 15 and change `image: <AWS::AccountId>.dkr.ecr.<AWS::Region>.amazonaws.com/xraydaemon:v1` to your flaskxray repositoryUri.

.  Deploy the Python Flask Application

	$ kubectl create -f templates/flaskyxray-deployment.yaml
	$ kubectl create -f templates/flaskyxray-service.yaml

.  Access the Python Flask web service

	$ kubectl get svc -owide 
	
	# look for the DNS name of the Load Balancer for **flaskxray-service** under the column **EXTERNAL-IP**
	$ curl http://<DNS NAME of ELB>
	Hello Flask

.	Navigate to AWS X-Ray Console to see your traces

image::images/flaskxray.png[flaskxray, 700]

== Tracing multiple services : A Node Express example

We will modify the Node Express applications https://github.com/arun-gupta/container-service-discovery[found here] for this exercise.

We will create our ECR docker image repositories. Next, we will build and push each of the 3 services to the ECR repository. (Note: You need **jq** https://stedolan.github.io/jq/download/[installed])
	
.  Login to ECR
	$ `aws ecr get-login --no-include-email`
	
. Build and push the `Greeter` service

	$ cd nodejs-microservices
	$ export greeter_repo=$(aws ecr create-repository --repository-name greeter| jq -r '.repository.repositoryUri')
	$ cd greeter && docker build -t $greeter_repo:v1 .
	$ docker push $greeter_repo:v1 && cd ..

. Build and push the `Name` service

	$ export name_repo=$(aws ecr create-repository --repository-name name| jq -r '.repository.repositoryUri')
	$ cd name && docker build -t $name_repo:v1 .
	$ docker push $name_repo:v1 && cd ..

. Build and push the `Webapp` service

	$ export webapp_repo=$(aws ecr create-repository --repository-name webapp| jq -r '.repository.repositoryUri')
	$ cd webapp && docker build -t $webapp_repo:v1 .
	$ docker push $webapp_repo:v1 && cd ..
	
	
.  Deploy the 3 services to Kubernetes
	
	$ cd ..
	$ sed -ie 's, '"<CHANGEME:NAMEIMAGE>"','" $name_repo:v1"',' templates/nodejs-microservices.yaml
	$ sed -ie 's, '"<CHANGEME:GREETIMAGE>"','" $greeter_repo:v1"',' templates/nodejs-microservices.yaml
	$ sed -ie 's, '"<CHANGEME:WEBAPPIMAGE>"','" $webapp_repo:v1"',' templates/nodejs-microservices.yaml
	$ kubectl create -f templates/nodejs-microservices.yaml --record

.  Access the web service

	$ kubectl get svc -owide 
	# look for the DNS name of the Load Balancer for **webapp-service** under the column **EXTERNAL-IP**
	$ curl http://<DNS NAME of ELB>

.	Navigate to AWS X-Ray Console to see your traces. You can see the time taken for `webapp` service to make the 2 calls to `greeter` and `name` services.

image::images/xray1.png[xray, 700]

image::images/xraytrace.png[xraytrace, 700]

== Delete all Kubernetes deployment and services
	$ kubectl delete -f templates/flaskyxray-deployment.yaml
	$ kubectl delete -f templates/flaskyxray-service.yaml
	$ kubectl delete -f templates/nodejs-microservices.yaml
	$ kubectl delete -f templates/daemonsetxray.yaml

== Delete images in ECR

	$	for n in webapp name greeter flaskxray xraydaemon
  		do
  			aws ecr delete-repository --repository-name $n  --force
		done