Automate AWS IAM for EKS

Otterize automates AWS IAM roles and policies for your AWS EKS workloads, all in Kubernetes.

In this tutorial, we will:

Optionally, spin up an EKS cluster.
Deploy a server pod that uploads files to S3, and a client pod that submits files to the server app.
Label the server pod, telling the credentials operator to link its Kubernetes ServiceAccount with an AWS IAM role created for it, such that AWS IRSA can recognize the pod.
Create a ClientIntents resource allowing the server pod to upload to S3, that tells the intents operator to update the previously-created role's policy.
See that the files have been uploaded successfully.

Prerequisites

Already have Otterize deployed with the IAM integration configured on your cluster? Skip to the tutorial.

1. Create an AWS EKS cluster

Before you start, you'll need an AWS EKS cluster. Any cluster will do; there are no special requirements or setup.

How to set up an AWS EKS cluster using eksctl

Run the following command to create your AWS cluster. Don't have eksctl? Install it now.

curl https://docs.otterize.com/code-examples/aws-iam-eks/cluster-config.yaml | eksctl create cluster -f -

Inspect eks-cluster.yaml contents

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: otterize-iam-eks-tutorial
  region: us-west-2
  version: "1.27"

iam:
  withOIDC: true

vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: true

addons:
  - name: vpc-cni
    version: 1.14.0
    attachPolicyARNs: #optional
      - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
    configurationValues: |-
      enableNetworkPolicy: "true"
  - name: coredns
  - name: kube-proxy

managedNodeGroups:
  - name: small-on-demand
    amiFamily: AmazonLinux2
    instanceTypes: [ "t3.large" ]
    minSize: 0
    desiredCapacity: 2
    maxSize: 6
    privateNetworking: true
    disableIMDSv1: true
    volumeSize: 100
    volumeType: gp3
    volumeEncrypted: true
    tags:
      team: "eks"

Don't forget to configure your kubeconfig for your cluster. If using the example cluster above, use this command:

aws eks update-kubeconfig --region us-west-2 --name otterize-iam-eks-tutorial

2. Deploy Otterize for AWS IAM

To deploy Otterize, head over to Otterize Cloud and:

Create a Kubernetes integration on the Integrations page, and follow the instructions. Make sure to enable enforcement mode for this tutorial. If you already have a Kubernetes cluster connected, skip this step.
Create an AWS IAM integration on the Integrations page.

If you are using the cluster from the previous step, the cluster name is otterize-iam-eks-tutorial and the region is us-west-2.

Once the AWS integration is configured, you'll be presented with a dialog to launch a CloudFormation template to configure the IAM roles for the Otterize operators. This enables the operators to manage IAM for the pods in your cluster, and configures AWS IRSA on your cluster, if it is not yet configured. This setup is required once per-cluster. This template can be found on GitHub as well.

After CloudFormation has configured your cluster, click Next and you'll be presented with the configuration for deploying Otterize. Since you now have the AWS integration enabled, you need to redeploy Otterize with the credentials operator enabled, and with configuration telling the operators which AWS roles they will AssumeRole to in order to operate.

See how to manually configure AWS IRSA on your cluster for Otterize

You may also manually configure your cluster.

Configure AWS IRSA on your cluster according to the official instructions from AWS.

Create IAM roles for the intents operator and credentials operator.

Create two roles, named '[your-cluster-name]-otterize-intents-operator' and '[your-cluster-name]-otterize-credentials-operator'.
Each of them should have a trust relationship with the intents-operator and credentials-operator service accounts on your cluster, and federated with your OpenIDConnect provider for the cluster. Here's an example:

{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Effect": "Allow",
        "Principal": {
            "Federated": "arn:aws:iam::353146681200:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/C90DC306FCDDFFD362B58B0D28D844CB"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
            "StringEquals": {
                "oidc.eks.us-west-2.amazonaws.com/id/C90DC306FCDDFFD362B58B0D28D844CB:sub": "system:serviceaccount:otterize-system:intents-operator-controller-manager"
            }
        }
    }
    ]
}

In this example, C90DC306FCDDFFD362B58B0D28D844CB is your OpenIDConnect provider ID. For the credentials-operator, substitute system:serviceaccount:otterize-system:intents-operator-controller-manager with system:serviceaccount:otterize-system:credentials-operator-controller-manager.

Create IAM policies for these two roles. They should grant access to all iam permissions in the account, and a subset of organizations and ec2 and eks permissions. Here's an example:

    Statement:
        - Effect: Allow
    Action:
        - "iam:*"
        - "organizations:DescribeAccount"
        - "organizations:DescribeOrganization"
        - "organizations:DescribeOrganizationalUnit"
        - "organizations:DescribePolicy"
        - "organizations:ListChildren"
        - "organizations:ListParents"
        - "organizations:ListPoliciesForTarget"
        - "organizations:ListRoots"
        - "organizations:ListPolicies"
        - "organizations:ListTargetsForPolicy"
        - "ec2:DescribeInstances"
        - "eks:DescribeCluster"
    Resource: "*"

Tutorial

Create an S3 bucket for the server to use

First, we need to pick an S3 bucket name. Because S3 buckets are globally unique, we will save the bucket name in an environment variable for use later.

export BUCKET_NAME=otterize-tutorial-bucket-`date +%s`
echo $BUCKET_NAME

aws s3api create-bucket --bucket $BUCKET_NAME --region us-west-2 --create-bucket-configuration LocationConstraint=us-west-2

Deploy the sample server and client

kubectl create namespace otterize-tutorial-iam
kubectl apply -n otterize-tutorial-iam -f https://docs.otterize.com/code-examples/aws-iam-eks/client-and-server.yaml
kubectl patch deployment -n otterize-tutorial-iam server --type='json' -p="[{\"op\": \"replace\", \"path\": \"/spec/template/spec/containers/0/env\", \"value\": [{\"name\": \"BUCKET_NAME\", \"value\": \"$BUCKET_NAME\"}]}]"

Expand to see the deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: server
  template:
    metadata:
      labels:
        app: server
    spec:
      serviceAccountName: server
      containers:
        - name: server
          imagePullPolicy: Always
          image: 'public.ecr.aws/e3b4k2v5/ekstutorial:server'
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: server
spec:
  type: ClusterIP
  selector:
    app: server
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
        - name: client
          imagePullPolicy: Always
          image: 'public.ecr.aws/e3b4k2v5/ekstutorial:client'
          ports:
            - containerPort: 80

View logs for the server - access denied

The server logs will show that it fails to upload files to the S3 bucket.

kubectl logs -f -n otterize-tutorial-iam deploy/server

2023/11/17 19:08:30 Couldn't upload file testfile.0.txt to otterize-tutorial-bucket-1700247949:testfile.0.txt.
Here's why: operation error S3: PutObject, https response error StatusCode: 403, RequestID: W9ZBNWPDFEMTBBS5, HostID: gsOlyxAmcUqFiYKzB7IwBoPia/Z2TDHEoaJjXT/qAxdduD4H17XDoGu21Pb7zeMms/YeThv2mJk=,
api error AccessDenied: Access Denied

Label the server pod to create an AWS IAM role

Label the server Pod so that the Otterize credentials operator creates an AWS IAM role for it and binds its Kubernetes ServiceAccount to the newly created role.

metadata:
  labels:
    credentials-operator.otterize.com/create-aws-role: "true"

To do this, we won't be labeling the Pod directly, but instead patching the template attribute of the Deployment we created earlier so that it updates the Pod.

kubectl patch deployment -n otterize-tutorial-iam server -p '{"spec": {"template":{"metadata":{"labels":{"credentials-operator.otterize.com/create-aws-role":"true"}}}} }'

An AWS IAM role was created

Let's inspect the created role:

aws iam list-roles --query 'Roles[?starts_with(RoleName, `otr-`) == `true`]'

In the output, you should see that a role was created, with an AssumeRolePolicyDocument that enables the server's ServiceAccount to AssumeRole.

    {
        "Path": "/",
        "RoleName": "otr-otterize-tutorial-iam.server@otterize-iam-eks-tutoria-ef91a7",
        "RoleId": "AROAVEOJOW5YM2CXSB4FJ",
        "Arn": "arn:aws:iam::353146681200:role/otr-otterize-tutorial-iam.server@otterize-iam-eks-tutoria-ef91a7",
        "CreateDate": "2023-11-21T12:03:42+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::353146681200:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/84E95D704D69DB40F4E4B6B6A6777CA3"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.us-west-2.amazonaws.com/id/84E95D704D69DB40F4E4B6B6A6777CA3:aud": "sts.amazonaws.com",
                            "oidc.eks.us-west-2.amazonaws.com/id/84E95D704D69DB40F4E4B6B6A6777CA3:sub": "system:serviceaccount:otterize-tutorial-iam:server"
                        }
                    }
                }
            ]
        },
    [...]
}
}

The Kubernetes ServiceAccount was annotated with the role ARN

The credentials operator automatically annotated the Kubernetes ServiceAccount for the server pod with the newly created role ARN.

Let's look at the service account:

kubectl get serviceaccount -n otterize-tutorial-iam server -o yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn:
      arn:aws:iam::353146681200:role/otr-otterize-tutorial-iam.server@otterize-iam-eks-tutoria-ef91a7
  name: server
  namespace: otterize-tutorial-iam

Apply intents to create the necessary IAM policy

By annotating the pod, we've created an IAM role. We now need to specify what we need to access, and the intents operator will create an IAM policy accordingly.

We will specify the following ClientIntents, granting the PutObject permission to the otterize-tutorial-bucket S3 bucket.

apiVersion: k8s.otterize.com/v1alpha3
kind: ClientIntents
metadata:
  name: server
spec:
  service:
    name: server
  calls:
    - name: arn:aws:s3:::otterize-tutorial-bucket-*/*
      type: aws
      awsActions:
        - "s3:PutObject"

To apply these intents, run the following command:

kubectl apply -n otterize-tutorial-iam -f https://docs.otterize.com/code-examples/aws-iam-eks/clientintents.yaml

The server can now upload files to S3!

Let's look at the server logs again to see that no more errors are being reported:

kubectl logs -f -n otterize-tutorial-iam deploy/server

{
  "status":200,
  "host":"server",
  "method":"POST",
  "uri":"/upload"
}

Let's list the contents of the S3 bucket:

aws s3 ls $BUCKET_NAME

2023-11-17 20:43:05         19 testfile.0.txt
2023-11-17 20:42:35         19 testfile.1.txt
2023-11-17 20:42:45         19 testfile.2.txt
2023-11-17 20:42:55         19 testfile.3.txt

What's next?

Try out some of the other quick tutorials to learn about how to use ClientIntents to manage network policies, Istio policies, PostgreSQL access, and more. You can use a single ClientIntents resource to specify all the access required for a pod.

Teardown

To remove the deployed examples run:

kubectl delete namespace otterize-tutorial-iam

To delete the cluster, if you created the one in this tutorial:

eksctl delete cluster -f cluster-config.yaml

To empty and delete the S3 bucket created for this tutorial:

aws s3 rm s3://otterize-tutorial-bucket --recursive
aws s3 rb s3://otterize-tutorial-bucket

Prerequisites​

1. Create an AWS EKS cluster​

2. Deploy Otterize for AWS IAM​

Tutorial​

Create an S3 bucket for the server to use​

Deploy the sample server and client​

View logs for the server - access denied​

Label the server pod to create an AWS IAM role​

An AWS IAM role was created​

The Kubernetes ServiceAccount was annotated with the role ARN​

Apply intents to create the necessary IAM policy​

The server can now upload files to S3!​

What's next?​

Teardown​