Skip to main content

Automate GCP IAM for GKE

Otterize automates GCP IAM roles and policies for your GCP GKE workloads, all in Kubernetes.

In this tutorial, we will:

  • Optionally, spin up a GKE cluster.
  • Deploy a server pod that uploads files to Google Cloud Storage, and a client pod that submits files to the server app.
  • Label the server pod, telling the credentials operator to link its Kubernetes ServiceAccount with a GCP service account created for it, such that GCP workload identity federation can recognize the pod.
  • Create a ClientIntents resource allowing the server pod to upload to GCS, that tells the intents operator to update the previously-created GCP service account with the relevant permissions.
  • See that the files have been uploaded successfully.

Prerequisites​

Already have Otterize deployed with the IAM integration configured on your cluster? Skip to the tutorial.

1. Create a GCP GKE cluster​

Before you start, you'll need an GCP GKE cluster. The cluster should have Workload identity federation and Config Connector installed.

How to set up a GKE cluster using gcloud CLI

Run the following commands to configure your project and create your cluster. Don't have gcloud? Install it now.

  1. Create a project and set the default gcloud configurations

    export PROJECT_NAME=otterize-gcp-demo
    export REGION=us-central1
    gcloud projects create $PROJECT_NAME
    gcloud config set project $PROJECT_NAME
    gcloud config set compute/region $REGION

  2. Enable the relevant APIS

    gcloud services enable container.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com

    Ensure that you have at least the following IAM roles: [roles/container.admin, roles/iam.serviceAccountAdmin]

  3. Create a new GKE cluster with workload identity and config connector enabled

    gcloud container clusters create otterize-iam-gke-tutorial \
    --release-channel regular \
    --addons ConfigConnector \
    --workload-pool=$PROJECT_NAME.svc.id.goog \
    --logging=SYSTEM \
    --monitoring=SYSTEM

Don't forget to configure your kubeconfig for your cluster. If using the example cluster above, use this command:

gcloud container clusters get-credentials otterize-iam-gke-tutorial

2. Deploy Otterize for GCP IAM​

To deploy Otterize, head over to Otterize Cloud and:

  1. Create a Kubernetes cluster on the Integrations page, and follow the instructions. Make sure to enable enforcement mode for this tutorial. If you already have a Kubernetes cluster connected, skip this step.

  2. Create an GCP IAM integration on the Integrations page.

If you are using the cluster from the previous step, the cluster name is otterize-iam-gke-tutorial and the region is us-central1.

Once the GCP integration is configured, you'll be presented with instructions for configuring your Otterize integration with GCP IAM support.

  • If you don't have a GCP service account for config connector, make sure to toggle "I don't have Config Connector on my cluster". This will tell terraform to create a GCP service account for config connector and give it the necessary permissions to manage GCP IAM. Choose this option if you deployed your GKE cluster using the instructions in the previous step.
  • If you have a GCP service account for config connector, keep the "I have Config Connector deployed with a GCP service account" toggle and provide the service account name. This will tell terraform to use the existing service account and give it the necessary permissions to manage GCP IAM.

After Terraform has configured your cluster, click Next and you'll be presented with the configuration for deploying Otterize. Since you now have the GCP integration enabled, you need to redeploy Otterize with GCP integration enabled flag, providing it the client ID for the managed identity created during the terraform installation.

See how to manually configure Config Connector on your cluster for Otterize

You may also manually configure your clusters config connector to be used with Otterize.

  1. Configure the GCP service account for Config Connector
    • Create a service account for Config Connector
        gcloud iam service-accounts create [CONFIG_CONNECTOR_SA_NAME]
    • Add the following permissions to the service account
        roles/iam.roleAdmin
        roles/iam.securityAdmin
        roles/iam.serviceAccountAdmin
        roles/iam.workloadIdentityUser
      You can use the following command to add permissions to the service account
        gcloud projects add-iam-policy-binding $PROJECT_NAME \
        --member="serviceAccount:[CONFIG_CONNECTOR_SA_NAME]@$PROJECT_NAME.iam.gserviceaccount.com" \
        --role="roles/iam.roleAdmin"
    • Bind the service account to workload identity
        gcloud iam service-accounts add-iam-policy-binding \
        [CONFIG_CONNECTOR_SA_NAME]@$PROJECT_NAME.iam.gserviceaccount.com \
        --member="serviceAccount:$PROJECT_NAME.svc.id.goog[cnrm-system/cnrm-controller-manager]" \
        --role="roles/iam.workloadIdentityUser"
  2. Apply the following YAML to your kubernetes cluster to finish the config connector configuration.
    apiVersion: core.cnrm.cloud.google.com/v1beta1
    kind: ConfigConnector
    metadata:
      name: configconnector.core.cnrm.cloud.google.com
    spec:
      mode: cluster
      googleServiceAccount: "[CONFIG_CONNECTOR_SA_NAME]@$PROJECT_NAME.iam.gserviceaccount.com"

Tutorial​

Create a GCS bucket for the server to use​

First, we need to pick a bucket name. Because GCS buckets are globally unique, we will save the bucket name in an environment variable for use later.

export BUCKET_NAME=otterize-tutorial-bucket-`date +%s`
echo $BUCKET_NAME
gcloud config set project $PROJECT_NAME
gsutil mb -c standard -l us-central1 gs://$BUCKET_NAME

Deploy the sample server and client​

kubectl create namespace otterize-tutorial-gcp-iam
kubectl apply -n otterize-tutorial-gcp-iam -f https://docs.otterize.com/code-examples/gcp-iam-gke/client-and-server.yaml
kubectl patch deployment -n otterize-tutorial-gcp-iam server --type='json' -p="[{\"op\": \"replace\", \"path\": \"/spec/template/spec/containers/0/env\", \"value\": [{\"name\": \"BUCKET_NAME\", \"value\": \"$BUCKET_NAME\"}]}]"
Expand to see the deployment YAML
apiVersion: v1
kind: Namespace
metadata:
  name: otterize-tutorial-gcp-iam
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: server
  namespace: otterize-tutorial-gcp-iam
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  namespace: otterize-tutorial-gcp-iam
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
        - name: client
          imagePullPolicy: Always
          image: 'public.ecr.aws/e3b4k2v5/gcp-tutorial:client'
          ports:
            - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: server
  namespace: otterize-tutorial-gcp-iam
spec:
  replicas: 1
  selector:
    matchLabels:
      app: server
  template:
    metadata:
      labels:
        app: server
    spec:
      serviceAccountName: server
      containers:
        - name: server
          imagePullPolicy: Always
          image: 'public.ecr.aws/e3b4k2v5/gcp-tutorial:server'
          ports:
            - containerPort: 80
          env:
            - name: BUCKET_NAME
              value: "otterize-demo-bucket"
---
apiVersion: v1
kind: Service
metadata:
  name: server
  namespace: otterize-tutorial-gcp-iam
spec:
  type: ClusterIP
  selector:
    app: server
  ports:
    - name: http
      port: 80
      targetPort: 80

View logs for the server - access denied​

The server logs will show that it fails to upload files to the GCS bucket.

kubectl logs -f -n otterize-tutorial-gcp-iam deploy/server
Error 403: <service_account> does not have storage.objects.create access to the Google Cloud Storage object.
Permission 'storage.objects.create' denied on resource (or it may not exist).

Label the server pod to create a GCP service account​

Label the server Pod so that the Otterize credentials operator creates a GCP service account and binds to the pods Kubernetes ServiceAccount.

metadata:
  labels:
    credentials-operator.otterize.com/create-gcp-sa: "true"

To do this, we won't be labeling the Pod directly, but instead patching the template attribute of the Deployment we created earlier so that it updates the Pod.

kubectl patch deployment -n otterize-tutorial-gcp-iam server -p '{"spec": {"template":{"metadata":{"labels":{"credentials-operator.otterize.com/create-gcp-sa":"true"}}}} }'

A GCP service account was created and bound to the server's Kubernetes ServiceAccount​

Let's inspect the created service account:

gcloud iam service-accounts list --filter="otr-"

The Kubernetes ServiceAccount was annotated with the role ARN​

The credentials operator automatically annotated the Kubernetes ServiceAccount for the server pod with the newly created GCP service account.

Let's look at the service account:

kubectl get serviceaccount -n otterize-tutorial-gcp-iam server -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account:
      otr-demo-cluster-otteri-3f630f@otterize-gcp-demo.iam.gserviceaccount.com
  name: server

Apply intents to create the necessary IAM policy​

By annotating the pod, we've created a GCP service account. We now need to specify what we need to access, and the intents operator will bind permissions accordingly.

We will specify the following ClientIntents, granting admin permission to the GCS bucket, and it's nested resources.

apiVersion: k8s.otterize.com/v1alpha3
kind: ClientIntents
metadata:
  name: server
  namespace: otterize-tutorial-gcp-iam
spec:
  service:
    name: server
  calls:
    - name: projects/_/buckets/otterize-tutorial-bucket*
      type: gcp
      gcpPermissions:
        - "storage.admin"

To apply these intents, run the following command:

kubectl apply -n otterize-tutorial-gcp-iam -f https://docs.otterize.com/code-examples/gcp-iam-gke/clientintents.yaml

The server can now upload files to GCS!​

Let's look at the server logs again to see that no more errors are being reported:

kubectl logs -f -n otterize-tutorial-gcp-iam deploy/server
{
  "status":200,
  "host":"server",
  "method":"POST",
  "uri":"/upload"
}

Let's list the contents of the S3 bucket:

gsutil ls gs://$BUCKET_NAME
gs://otterize-tutorial-bucket-1710338230/testfile.0.txt
gs://otterize-tutorial-bucket-1710338230/testfile.1.txt
gs://otterize-tutorial-bucket-1710338230/testfile.2.txt
gs://otterize-tutorial-bucket-1710338230/testfile.3.txt

What's next?​

Try out some of the other quick tutorials to learn about how to use ClientIntents to manage network policies, Istio policies, PostgreSQL access, and more. You can use a single ClientIntents resource to specify all the access required for a pod.

Teardown​

To remove the deployed examples run:

kubectl delete namespace otterize-tutorial-gcp-iam

To delete the cluster, if you created the one in this tutorial:

gcloud container clusters delete otterize-iam-gke-tutorial

To empty and delete the GCS bucket created for this tutorial:

gsutil -m rm -r gs://$BUCKET_NAME