# Centralized logging for multi-tenant applications on Amazon EKS

In this example, we will showcase how to build a centralized logging solution for multi-tenant environments where applications that belong to different teams or customers run in a shared Amazon EKS cluster.

## Components

Fluent Bit DaemonSet to collect, modify, and enrich logs from applications, and publish the logs to Amazon Managed Streaming for Kafka (Amazon MSK).

Amazon MSK to forward log events to various destination and as a buffering layer to avoid indexing pressure in Amazon OpenSearch. This layer will provide tenant isolation and improve resilience of the solution.

Amazon OpenSearch to monitor, visualize, and analyze logs. OpenSearch is a distributed, community-driven, Apache 2.0-licensed, 100% open-source search and analytics suite used for a broad set of use cases like real-time application monitoring, log analytics, and website search.

Terraform by HashiCorp, is an infrastructure as code tool similar to AWS CloudFormation to provision and manage infrastructure on AWS.

## Features

Security and compliance - This solution provides data isolation for logs ingested and stored so that each tenant can only access their own logs. It also can help you to meet your compliance requirements, for example anonymizing personally identifiable information (PII) in logs which is required by GDPR.

Business and technical insights -  Using OpenSearch Dashboards, you can produce business insights such as geographical distribution of customers or popularity of a product over time. Software engineers can also use this solution to troubleshoot an issue, or to create metrics and alarms to proactively notify the application owners of any issues.

Log routing - You can forward logs to various destinations for different purposes such as logs archival, cold logs analysis, and third party observability or security information and event management (SIEM) solutions such as Splunk or Datadog.

## Architecture

In this example, we assumed that you use Kubernetes Namespaces to divide cluster resources between multiple users or tenants as a foundation for multi-tenancy. Furthermore, during log ingestion and storage, we used Kafka topics and OpenSearch indices to isolate logs generated by each tenant. This approach enables you to control who can access logs in any stage.

In the following diagram Product, Order, and Payment are example of tenants which isolated by Namespaces in Kubernetes, Topics in Kafka, and Indices in OpenSearch.

![Architecture](Ref-Architecture.png?raw=true "Title")

### Fluent Bit configuration

We used the Lua filter plugin to add a key `namespace` with value of Kubernetes namespace name prefixed by `logs_`. This key is subsequently used in the Kafka output plugin to define Kafka topic name dynamically. Therefore, all logs belong to a Kubernetes namespace will be grouped by a Kafka topic.

If you need a custom parser for your application logs, Fluent Bit allows you to suggest a pre-defined parser by annotating your application Pod definition using `fluentbit.io/parser: <parser-name>`. You can also completely opt out of logging for any of your Pods using `fluentbit.io/exclude: "true"` annotation. For more information, see [Fluent Bit - Kubernetes Annotations](https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#kubernetes-annotations).

### Multi-tenancy on OpenSearch

All logs from a Kubernetes namespace will be store in a separate index with `logs_<namespace>` naming convention. With this approach, you can define proper access control at Kubernetes namespace or OpenSearch index level for each tenant. For more information see [Storing Multi-Tenant SaaS Data with Amazon OpenSearch Service](https://aws.amazon.com/blogs/apn/storing-multi-tenant-saas-data-with-amazon-opensearch-service/)

## Prerequisites

* An Amazon S3 bucket as a Terraform backend to store Terraform state data files.
* An IAM role, or an IAM user principal with required privileges to create resources such as Amazon VPC, Amazon S3 bucket, Amazon EKS cluster, Amazon MSK cluster, and Amazon OpenSearch cluster.
* A service-linked role to create Amazon OpenSearch VPC domains. You can run the following command to create the required service-linked role if the role doesn't exist.

    ```bash
    aws iam get-role --role-name "AWSServiceRoleForAmazonOpenSearchService" || aws iam create-service-linked-role --aws-service-name "opensearchservice.amazonaws.com"
    ```

* The following tools installed on a machine.
  * Kubectl - for more information, see [Installing or updating kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html).
  * Terraform - for more information, see [Installing Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli).
  * Optionally, Apache Kafka client libraries and tools - for more information, see [Create a topic](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html).

## Instructions

1. Clone the repository

    ```bash
    git clone https://github.com/aws-samples/amazon-eks-fluent-logging-examples.git
    ```

2. Change directory into terraform directory

    ```bash
    cd amazon-eks-fluent-logging-examples/examples/eks-msk-opensearch/terraform
    ```

3. Update `provider.tf` file with your S3 bucket name, key, and region for Terraform to store its state.

    Backend example

    ```hcl
    terraform {
    backend "s3" {
        bucket = "yourbucket"
        key = "path/to/your/key"
        region = "us-east-1"
    }
    ```

4. Optionally, configure `variables.tf` file to add Kubernetes namespaces you need. You can also use `enable_logs_to_es` boolean attribute to enable or disable logging for each namespace. For example we are creating 4 namespaces with logging enabled. "logging" namespace is being used to deploy fluent-bit.

    ```hcl
    variable "namespaces" {
    description = "K8s namespaces to create"
    type = list(object({
        name              = string
        enable_logs_to_es = bool
    }))
    default = [
        {
        "name" : "payment",
        "enable_logs_to_es" = true,
        },
        {
        "name" : "order",
        "enable_logs_to_es" = true,
        },
        {
        "name" : "product",
        "enable_logs_to_es" = true,
        },
        {
        "name" : "logging",
        "enable_logs_to_es" = true,
        }
    ]
    }
    ```

5. Initialize the Terraform working directory.

    ```hcl
    terraform init
    ```

6. Create an execution plan to verify the resources that Terraform will create for you.

    ```hcl
    terraform plan
    ```

7. Executes the actions proposed in a Terraform plan to create the infrastructure. This command will ask you for the OpenSearch domain admin password which you need later to access the OpenSearch dashboard.

    ```hcl
    terraform apply
    ```

8. Update the kubeconfig file to interact with your newly created eks cluster.The name of cluster is choosen from variables in variables.tf in pattern of eks-<org>-<env>-<regionshort>

    ```bash
    aws eks update-kubeconfig --region <aws-region> --name <clustername>
    
    ```

9. Create two sample nginx deployment in `product` and 'order' namespace to generate sample logs.

    ```bash
    kubectl config set-context --current --namespace=product
    kubectl apply -f ../example-deployment.yaml
    kubectl get svc nginx-service-loadbalancer
 
    kubectl config set-context --current --namespace=order
    kubectl apply -f ../example-deployment.yaml
    kubectl get svc nginx-service-loadbalancer
    ```

10. Take note of the load balancer urls from above and open them in two seperate browser windows then hit it few times to generate sample logs.

11. Verify `logs_example` Kafka topic is created, and read message from the topic using the following commands.

    ```bash
    ./bin/kafka-topics.sh --bootstrap-server=<list of your brokers>  --list
    ./bin/kafka-console-consumer.sh --bootstrap-server <list of your brokers> --topic product
    ./bin/kafka-console-consumer.sh --bootstrap-server <list of your brokers> --topic order
    ```

12. Login to OpenSearch dashboard to visualize our logs.

    > **Note:** Because this solution creates an OpenSearch Service domain within a VPC, your computer must be able to connect to the VPC. This connection often takes the form of a VPN, transit gateway, managed network, or proxy server. You can't directly access your domains from outside the VPC. For more information, see [Using OpenSearch Dashboards with Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/dashboards.html)
    
    
**Note:**
As an alternative to 'nest' filter in fluent-bit yaml, following lua script filter can also be used to set the custom topic name with logs_ prefix.
    
    [FILTER]
        Name                lua
        Match               kube.*
        call                set_topic
        code                function set_topic(tag, timestamp, record) if record["kubernetes"]["namespace_name"] ~= nil then record["namespace"] = "logs_" .. record["kubernetes"]["namespace_name"] end return 2, timestamp, record end