Skip to main content

AWS resource mapping & IAM policy generation

Many production Kubernetes workloads rely on cloud resources, like S3 Buckets, RDS databases, and Lambda functions. In this tutorial, we will look at how Otterize provides visibility into the AWS resources called by your workloads.

In this tutorial, we will:

  • Set up an EKS cluster.
  • Deploy two Lambda functions.
  • Deploy a server pod that retrieves a joke (as in, a string containing a joke ;) from a Lambda, provides a review, and posts the review to another Lambda.
  • Automatically detect and view the Lambda function calls in Otterize.

By the end, you'll know how to map Kubernetes workloads alongside their dependent AWS resources using Otterize.

Prerequisites​

CLI tools​

We will need the following CLI tools to set up our cluster and deploy our scripts.

  1. AWS CLI. You will also need credentials within the target account with permissions to work with EKS clusters, IAM, CloudFormation, and Lambda functions.
  2. eksctl

Create an EKS cluster​

Already have Otterize deployed with the IAM integration configured on your cluster? Skip to the tutorial.

Begin by creating an EKS cluster for pod deployment using eksctl with the YAML configuration below:

curl https://docs.otterize.com/code-examples/aws-visibility/eks-cluster.yaml | eksctl create cluster -f -
Inspect eks-cluster.yaml contents
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: otterize-tutorial-aws-visibility
  region: us-west-2

iam:
  withOIDC: true

vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: true

addons:
  - name: vpc-cni
    version: 1.14.0
    attachPolicyARNs:
      - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
    configurationValues: |-
      enableNetworkPolicy: "true"
  - name: coredns
  - name: kube-proxy

managedNodeGroups:
  - name: x86-al2-on-demand
    amiFamily: AmazonLinux2
    instanceTypes: [ "t3.large" ]
    minSize: 0
    desiredCapacity: 2
    maxSize: 6
    privateNetworking: true
    disableIMDSv1: true
    volumeSize: 100
    volumeType: gp3
    volumeEncrypted: true
    tags:
      team: "eks"

Next, update kubeconfig to link it with the new cluster:

aws eks update-kubeconfig --name otterize-tutorial-aws-visibility --region 'us-west-2'

Enable AWS Visibility with Otterize Installation​

To provide visibility, we will need to install Otterize in our cluster, and we will want to enable AWS IAM roles for service accounts (IRSA) on our cluster. We can quickly enable this using a cloudformation template on the Otterize Cloud Integrations page.

  1. Install Otterize If you don't have a connected Kubernetes cluster, create one via Integrations page and follow the setup instructions for Kubernetes. Skip if your cluster is already connected.

  2. Integrate AWS with Otterize Cloud To begin the integration with AWS, visit the Integrations page. Once there, you will be asked for information to help populate a CloudFormation template we will use to set up roles and policies for the Otterize deployment in our cluster.

If you created the EKS cluster above, the cluster name would be otterize-tutorial-aws-visibility, and the region would be us-west-2.

Once the information is provided, a Launch Cloudformation button will take you to the AWS Console to deploy the cloudformation script. This script will install IRSA within your EKS cluster and enable Otterize Cloud to manage intents.

After IRSA is enabled in your cluster, you need to redeploy Otterize with the AWS credential operator and AWS visibility enabled. In Otterize Cloud, click the Next button to see the updated Helm commands. AWS Visibility is not enabled by default. Before executing the revised configuration, you will need to set an additional flag at the end of the command:

--set networkMapper.aws.visibility.enabled=true

Tutorial​

Having configured our environment, we'll deploy AWS resources, authorize pod access using ClientIntents, and monitor access in Otterize Cloud.

Deploy two Lambda functions​

First, we will deploy two Lambda functions (DadJokeLambdaFunction and FeedbackLambdaFunction). These services will work alongside our server pod to generate a humor training dataset. This works by receiving a joke from the DadJokeLambdaFunction, the server pod reviewing the joke, and then sending the feedback to the FeedbackLambdaFunction.

We can deploy the lambda functions and their required roles with the following command:

curl https://docs.otterize.com/code-examples/aws-visibility/cloudformation.yaml -o template.yaml && \
aws cloudformation deploy --template-file template.yaml --stack-name OtterizeTutorialJokeTrainingStack --capabilities CAPABILITY_IAM --region 'us-west-2'
Inspect CloudFormation YAML
AWSTemplateFormatVersion: '2010-09-09'
Description: >-
  Stack creates two Lambda functions for dad jokes and feedback, plus the user and related policies

Resources:
  DadJokeLambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: 'DadJokeLambdaExecutionPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: 'arn:aws:logs:*:*:*'
              - Effect: Allow
                Action:
                  - 'execute-api:Invoke'
                Resource: '*'

  DadJokeLambda:
    Type: 'AWS::Lambda::Function'
    Properties:
      Handler: 'index.handler'
      Role: !GetAtt DadJokeLambdaExecutionRole.Arn
      Runtime: 'nodejs20.x'
      Code:
        ZipFile: |
          const https = require('https');
          exports.handler = async (event) => {
            return new Promise((resolve, reject) => {
              const options = {
                hostname: 'icanhazdadjoke.com',
                method: 'GET',
                headers: { 'Accept': 'application/json' }
              };
              const req = https.request(options, (res) => {
                let data = '';
                res.on('data', (chunk) => {
                  data += chunk;
                });
                res.on('end', () => {
                  resolve({
                    statusCode: 200,
                    body: data,
                    headers: { 'Content-Type': 'application/json' },
                  });
                });
              });
              req.on('error', (e) => {
                reject({
                  statusCode: 500,
                  body: 'Error fetching dad joke',
                });
              });
              req.end();
            });
          };
      Timeout: 10

  FeedbackLambda:
    Type: 'AWS::Lambda::Function'
    Properties:
      Handler: 'index.handler'
      Role: !GetAtt DadJokeLambdaExecutionRole.Arn
      Runtime: 'nodejs20.x'
      Code:
        ZipFile: |
          exports.handler = async (event) => {
              const payload = JSON.parse(event.body);

              const joke = payload.joke;
              const isFunny = payload.funny;

              console.log("Joke:", joke);
              console.log("Is it funny?", isFunny);

              return {
                  statusCode: 200,
                  body: JSON.stringify({ message: "Feedback received, Adding To Training Set" }),
              };
          };
      Timeout: 10

Outputs:
  DadJokeLambdaFunction:
    Description: "Dad Joke Lambda Function ARN"
    Value: !GetAtt DadJokeLambda.Arn

  FeedbackLambdaFunction:
    Description: "Feedback Lambda Function ARN"
    Value: !GetAtt FeedbackLambda.Arn


Deploy server with access to Lambda functions​

Now that our Lambdas are deployed, we want to deploy our server pod within our cluster and point it to our two Lambda functions. In the commands below, we will create a configmap to hold our functions ARNs and pass the map into our deployment YAML.


kubectl create namespace otterize-tutorial-aws-visibility

DAD_JOKE_LAMBDA_ARN=$(aws cloudformation describe-stacks --region 'us-west-2' --stack-name OtterizeTutorialJokeTrainingStack --query "Stacks[0].Outputs[?OutputKey=='DadJokeLambdaFunction'].OutputValue" --output text)
FEEDBACK_LAMBDA_ARN=$(aws cloudformation describe-stacks --region 'us-west-2' --stack-name OtterizeTutorialJokeTrainingStack --query "Stacks[0].Outputs[?OutputKey=='FeedbackLambdaFunction'].OutputValue" --output text)

kubectl create configmap lambda-arns \
  --from-literal=dadJokeLambdaArn=$DAD_JOKE_LAMBDA_ARN \
  --from-literal=feedbackLambdaArn=$FEEDBACK_LAMBDA_ARN \
  -n otterize-tutorial-aws-visibility

kubectl apply -n otterize-tutorial-aws-visibility -f https://docs.otterize.com/code-examples/aws-visibility/all.yaml
Inspect deployment YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: joketrainer
  namespace: otterize-tutorial-aws-visibility
spec:
  replicas: 1
  selector:
    matchLabels:
      app: joketrainer
  template:
    metadata:
      labels:
        app: joketrainer
        network-mapper.otterize.com/aws-visibility: "true"
        credentials-operator.otterize.com/create-aws-role: "true"
    spec:
      containers:
        - name: joketrainer
          image: otterize/aws-visibility-tutorial:latest
          env:
            - name: DAD_JOKE_LAMBDA_ARN
              valueFrom:
                configMapKeyRef:
                  name: lambda-arns
                  key: dadJokeLambdaArn
            - name: FEEDBACK_LAMBDA_ARN
              valueFrom:
                configMapKeyRef:
                  name: lambda-arns
                  key: feedbackLambdaArn

Inspecting our deployment YAML, you will see we have added two labels to our pod. The first network-mapper.otterize.com/aws-visibility informs the network mapper to identify AWS API calls, and the credentials-operator.otterize.com/create-aws-role that drives the credentials operator to create a role specifically for this pod that will be used for our intents.

Once our pod is deployed, we can inspect the logs and see that we cannot access the Lambda functions.

kubectl logs -f -n otterize-tutorial-aws-visibility deploy/joketrainer

Sample output:

invoke error, operation error Lambda: Invoke, https response error StatusCode: 403, RequestID: a3bab063-dfb0-49e3-b466-0069807c56fa, api error AccessDeniedException: User: arn:aws:sts::12345678910:assumed-role/otr-otterize-tutorial-aws-visibility.default@otterize-tut-ecfd9d/12345678910 is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:12345678910:function:OtterizeTutorialJokeTrainingStack-DadJokeLambda-dnNYqlwipYxG because no identity-based policy allows the lambda:InvokeFunction action

Applying Intents​

We can apply an intent for our pod to be able to call the Lambda functions created by our cloudformation stack.

kubectl apply -n otterize-tutorial-aws-visibility -f https://docs.otterize.com/code-examples/aws-visibility/intents.yaml
apiVersion: k8s.otterize.com/v1alpha3
kind: ClientIntents
metadata:
  name: server
spec:
  service:
    name: joketrainer
  calls:
    - name: arn:aws:lambda:us-west-2:*:function:OtterizeTutorialJokeTrainingStack-*
      type: aws
      awsActions:
        - "lambda:InvokeFunction"

We can now recheck the logs to ensure that the pod is running:

kubectl logs -f -n otterize-tutorial-aws-visibility deploy/joketrainer

Example output:

Joke: People saying 'boo! to their friends has risen by 85% in the last year.... That's a frightening statistic.
Sending Feedback of Funny?: Yes
Joke: Have you ever heard of a music group called Cellophane? They mostly wrap.
Sending Feedback of Funny?: Yes
Joke: What did Yoda say when he saw himself in 4K? "HDMI"
Sending Feedback of Funny?: No

Visualize Relationships​

The Otterize network mapper inspects pods with the network-mapper.otterize.com/aws-visibility: true label. For the labeled pods, the network mapper will identify AWS API calls made by that pod and determine which resources and actions are being used. This information is shown on the Access graph.

In the Access graph screenshot below, you’ll see 4 AWS resources associated with our joketrainer pod: DadJokeLambdaFunction, FeedbackLambdaFunction, the role assumed by our server pod, and our wildcard intent definition. This wildcard definition matches any Lambdas created by our cloudformation stack. These types of wildcard definitions can be helpful for AWS Resources with dynamic ARN names as you move across staging and production deployments. Still, they open up a security space that could be overly permissive for some environments. Otterize makes deploying with a wildcard definition easy and then applying more stringent authorization without disrupting any services.

Otterize Cloud AWS Visibility Example

What's Next​

Now that we've discovered the AWS resources used within a Kubernetes workload, you can learn more about how you can manage access to these resources with Otterize in the Automate AWS IAM for EKS tutorial.

Cleanup​

To remove the deployed example:

kubectl delete namespace otterize-tutorial-aws-visibility

To remove the Lambda functions:

aws cloudformation delete-stack --stack-name OtterizeTutorialJokeTrainingStack

To remove the EKS cluster:

eksctl delete cluster --name otterize-tutorial-aws-visibility --region us-west-2