Deploy Komiser to EKS Cluster - Multi Account Configuration

Introduction

As organizations expand their cloud footprints in the dynamic landscape of cloud computing, understanding and optimizing resource usage becomes crucial to maintaining efficiency and cost-effectiveness. Amazon Web Services (AWS) provides an abundance of services and resources, but optimizing their usage across multiple AWS accounts presents a formidable challenge.

Fortunately, Komiser offers a powerful solution to address this challenge by seamlessly collecting comprehensive resource data across your organisation's AWS accounts. With Komiser, you can gain deeper insights into resource consumption and expenditure across different AWS accounts, empowering you to make informed decisions and optimise your cloud infrastructure efficiently, and that is what we cover in this article.

In this tutorial, we will walk through the step-by-step process of deploying Komiser onto an EKS cluster, enabling it to access and aggregate resource data from two different AWS accounts.

Whether you're a DevOps engineer responsible for managing AWS resources or a cloud architect looking to optimise costs across different AWS accounts, this tutorial will equip you with the skills needed to make informed decisions and drive cost savings in your AWS infrastructure.

Architecture Overview

Here is an architectural overview of our aim for today. We’ll be working with two different AWS accounts:

1. Admin account
2. Dev account

Note: Here we are working with two completely different AWS accounts (which can be under the same organisation, but need not be) and not different IAM users under the same account!

Pre-requisites

Before you begin this tutorial, you’ll need the following:

1. Komiser CLI installed on your machine. (Make sure to upgrade to the latest version if already have Komiser installed on your machine)
2. kubectl installed on your machine, to interact with the EKS cluster.
3. Make sure to have eksctl installed on your machine, which we’ll be using to easily bootstrap an EKS cluster.
4. The latest version of AWS CLI installed on your machine.
5. Make sure to have Helm installed on your machine as we’ll be using a Helm chart to deploy Komiser to our EKS cluster.
6. Lastly, two different AWS accounts with properly configured credentials.

Step 1 - Setting up an EKS cluster using eksctl

Let us first get our EKS cluster up and running in our "Admin account". For this purpose, we’ll be using eksctl, which is the official CLI tool for creating and managing clusters on EKS.

Note: You can also create a new cluster using the AWS Management Console, by logging in to your Admin’s AWS account.

Use the following command to create a new EKS cluster using Kubernetes 1.27:

eksctl create cluster --name komiser-cluster --region us-west-1 --version=1.27

NBe sure to specify the version of Kubernetes using --version flag to always use the latest EKS versions available.

This will setup a simple two nodes EKS cluster, which you can then access using kubectl on your machine.

Note: If you have multiple clusters running on your machine, you may use a tool like kubectx  to easily switch between different clusters.

Step 2 - Create IAM OIDC provider in ADMIN account

To establish a trust relationship between our EKS cluster and the Admin IAM role (created later) for managing access to our cluster, we’ll be creating a new IAM OIDC identity provider.

The detailed steps to follow for creating a new IAM OIDC provider for your cluster are mentioned in the documentation.

Step 3 - Register ADMIN OIDC provider in DEV account

To establish a trust relationship between our EKS cluster in the Admin account and the Dev IAM role (created later), we’ll be registering the IAM OIDC provider (created in Step 2) with our Dev AWS account.

Step 1:

Log in to your Dev AWS account’s management console.

Step 2:

The detailed steps to follow for registering the IAM OIDC with the Dev account are mentioned in the documentation.

Step 4 - Create a new ADMIN IAM role

Below are the main steps to create and properly configure a new Admin IAM role:

Step 1:

Let us first generate the trust relationship policy for our IAM role with the service account. Create a new "trust.json" file in your local machine and paste the following code block:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ADMIN_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER_URL}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          ":aud": "sts.amazonaws.com",
          ":sub": "system:serviceaccount::${NAMESPACE}"
        }
      }
    }
  ]
}

1. Make sure to substitute ${} placeholders with the correct values accordingly.

2. Replace ${NAMESPACE} with the namespace where you want to deploy the Komiser helm chart later!

Step 2:

Use the following AWS CLI command to create a new role named "komiser-admin", with the above trust relationship policy:

aws iam create-role --role-name komiser-admin --assume-role-policy-document file://trust.json --description "admin role for komiser"

‍Step 3:

Using the AWS management console, create a new custom policy using the recommended Komiser policy.

Detailed steps for this are also mentioned in the documentation.

Step 4:

Use the following AWS CLI command to attach the "komiser-admin" role with the custom Komiser policy:

aws iam attach-role-policy --role-name komiser-admin --policy-arn=${POLICY_ARN}

Now our Admin IAM role is properly configured. Although, we’ll be visiting this again to make some changes once we create our Dev IAM role.

Step 5 - Create a new DEV IAM role

Below are the main steps to create and properly configure a new Dev IAM role for our Dev AWS account:

Step 1:

Using the AWS management console, create a new custom policy using the recommended Komiser policy.

Detailed steps for this are also mentioned in the documentation.

Step 2:

Let us first generate the trust relationship policy for our Dev IAM role with the ADMIN IAM role (created before). Create a new trust-dev.json file in your local machine and paste the following code block:

{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Principal": {
                 "AWS": "${ADMIN_IAM_ROLE_ARN}"
             },
             "Action": "sts:AssumeRole",
             "Condition": {}
         }
     ]
 }

Step 3:

Use the following AWS CLI command to create a new role named komiser-dev, with the above trust relationship policy:

aws iam create-role --role-name komiser-dev --assume-role-policy-document file://trust-dev.json --description "dev role"

Step 4:

Use the following AWS CLI command to attach the komiser-dev role with the custom Komiser policy:

aws iam attach-role-policy --role-name komiser-dev --policy-arn=${POLICY_ARN}

Great! Now we have configured our Dev IAM role and established a one-way trust relationship between the the Dev account and the Admin account.

Step 6: Add “Assume Dev role” policy to ADMIN IAM

To establish a two-way trust between the Dev account and the Admin account, we’ll need to modify the ADMIN IAM role’s policy to also assume the Dev IAM role (created before). This can be done by adding the below statement to the existing Admin IAM policy (custom Komiser policy that we created before):

{
	   "Sid": "6",
	   "Effect": "Allow",
	   "Action": "sts:AssumeRole",
	   "Resource": "arn:aws:iam::${DEV_ACCOUNT_ID}:role/${DEV_IAM_ROLE}"
}

Great! We have successfully configured both the ADMIN and the Dev IAM role and established a two-way trust relationship between the Admin and our Dev account 🎉

Step 7: Helm Chart Configuration

To deploy Komiser on our EKS cluster, we’ll be modifying an existing Helm chart which can be found in the GitHub repo.

Let us go step-by-step to configure this Helm chart according to our use case:

Step 1:

Add the ADMIN IAM role to the ServiceAccount.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: komiser
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${ACCOUNT_ID}:role/komiser-admin

Step 2:

Add the following configmap.yaml file in the /templates directory, which we are using to input the config.toml file for Komiser:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-configmap
  annotations:
    meta.helm.sh/release-name: komiser
    meta.helm.sh/release-namespace: ${NAMESPACE}
  labels:
    app.kubernetes.io/managed-by: Helm
  namespace: ${NAMESPACE}
data:
 config.toml: |-
   [[aws]]
   name="Admin Account"
   source="CREDENTIALS_FILE"
   path="/path/to/credentials/file"
   profile="Kunal" # Required if CREDENTIALS_FILE is set

   [[aws]]
   name="Dev Account"
   source="CREDENTIALS_FILE"
   path="/path/to/credentials/file"
   profile="Dev-User" # Required if CREDENTIALS_FILE is set

   [sqlite]
   file="komiser.db"

Make sure the replace ${NAMESPACE} with the correct namespace we configured before, while creating the trust relationship between the ADMIN IAM role and the service account.

Step 3:

Use the deployment manifest file below to mount the configMap we created above:

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: komiser-deploy
spec:
  selector: 
    matchLabels: 
      app: komiser
  template: 
     metadata: 
      name: komiser
      labels: 
        app: komiser
     spec:
      serviceAccountName: komiser
      volumes:
      - name: test-volume
        configMap:
          name: aws-configmap
      containers: 
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command: ["komiser","start","--config","/root/.aws/config.toml"]
          env:
          - name: AWS_DEFAULT_REGION
            value: "{{ .Values.aws.region }}"
          - name: AWS_CONFIG_FILE
            value: /root/.aws/credentials
          volumeMounts:
          - name: test-volume
            mountPath: /root/.aws/

1. Make sure not to change the mount path or internal volume path here. The paths should match the example above.

2. The config.toml file will be mounted as a volume at the location: /root/.aws/. Therefore, make sure to provide the same path in the komiser start command for the container.

Step 4:

To access the Komiser dashboard, we’ll modify the existing service.yaml file to use the NodePort type service for the komiser pod:

apiVersion: v1
kind: Service
metadata:
  name: komiser-svc
spec:
  ports:
    - name: http
      port: 80
      targetPort: 3000
  type: NodePort    
  selector:
    app: komiser

Step 5:

Lastly, update the values.yaml file to use the default AWS region to be used (same as the one we used to create our EKS cluster):

image:
  repository: tailwarden/komiser
  tag: 3.1.1  
  pullPolicy: IfNotPresent
aws:
  region: "us-west-1"

Step 8: Deploy Komiser on our EKS Cluster

Below are the steps required to deploy Komiser on our EKS cluster and accessing the Komiser dashboard.

Step 1:

While in the root directory of the helm repo, use the following command to install the Komiser Helm chart on our EKS cluster:

helm install komiser .

Step 2:

If everything is properly configured, you’ll see the Komiser pod successfully running on your EKS cluster.

Step 3:

This is an important step! To access the Komiser dashboard, we’ll need to use port forwarding to forward the traffic from our NodePort to the service port.

You can view the service that was created using the following command:

Now, use the following command to port-forward the traffic from the NodePort to the service port:

kubectl port-forward svc/komiser-svc NODE_PORT:80

Finally, you’ll be able to access the Komiser dashboard at http://localhost:NODE_PORT 🎉

If you have been following along so far, then Congratulations 🎉 as we have successfully configured Komiser to monitor the cloud resources from two different AWS accounts!

Heading over to the Inventory section, you can apply the desired filters to filter out resources being used from both the AWS accounts separately, as shown below. (a gif here).

Conclusion

In conclusion, Komiser simplifies AWS resource optimization by providing valuable insights across multiple accounts. By following the steps in this tutorial, you've learned how to deploy Komiser on your EKS cluster, enabling smarter decision-making and cost savings across different AWS accounts.

Whether you're managing cloud resources or overseeing cloud infrastructure, Komiser equips you to make the most of your AWS investment.

‍