Skip to main content

Kubernetes mTLS

Otterize can automatically provision mTLS credentials using Kubernetes pod identities and integrating with SPIFFE/SPIRE. Here we document how to generate mTLS credentials, how to consume them in a variety of languages, and how to verify them if needed.

Provisioning mTLS credentials​

To provision mTLS credentials for a client pod with Otterize, make the following 3 simple changes to its Kubernetes spec:

  1. Generate credentials: add the credentials-operator.otterize.com/tls-secret-name annotation, which tells the Otterize credentials operator to generate mTLS credentials, and to store them in a Kubernetes secret whose name is the value of this annotation.
  2. Expose credentials in a volume: add a volume containing this secret to the pod.
  3. Mount the volume: mount the volume in every container in the pod.

Here is the general structure for such a spec:

spec:
  template:
    metadata:
      annotations:
        # 1. Generate credentials as a secret called "client-credentials-secret":
        credentials-operator.otterize.com/tls-secret-name: client-credentials-secret
        ...
    spec:
      volumes:
        # 2. Create a volume containing this secret:
        - name: otterize-credentials
          secret:
            secretName: client-credentials-secret
        ...
      containers:
        - name: client
          ...
          volumeMounts:
            # 3. Mount volume into container
            - name: otterize-credentials
              mountPath: /var/otterize/credentials
              readOnly: true
tip

For the complete list of annotation parameters please consult the Otterize credentials operator documentation.

tip

Certificates are automatically refreshed before expiring. We recommend loading certificates each time before using them, to avoid using expired certificates.

Using mTLS credentials​

The generated mTLS credentials can be used by services directly in their native programming languages via common SDKs. The following examples showcase how to use the generated mTLS credentials for mTLS between:

  • HTTP servers and their clients
  • Kafka brokers and their clients

HTTP​

Client​

const fs = require('fs');
const https = require('https');

const options = {
    hostname: 'server.otterize-tutorial-mtls/hello',
    port: 443,
    path: '/hello',
    method: 'GET',
    cert: fs.readFileSync('/var/otterize/credentials/cert.pem'),
    key: fs.readFileSync('/var/otterize/credentials/key.pem'),
    ca: fs.readFileSync('/var/otterize/credentials/ca.pem')
}

const req = https.request(
    options,
    res => {
        res.on('data', function (data) {
            console.log(data)
        });
    }
);

req.end();

Server​

const https = require(`https`);
const fs = require(`fs`);

const options = {
    key: fs.readFileSync('/var/otterize/credentials/key.pem'),
    cert: fs.readFileSync('/var/otterize/credentials/cert.pem'),
    ca: fs.readFileSync('/var/otterize/credentials/ca.pem'),
    requestCert: true
};

https.createServer(
    options,
    (req, res) => {
        const peerCert = req.connection.getPeerCertificate();
        const ownCert = req.connection.getCertificate();
        console.log("Received request:");
        console.log(peerCert.subject.CN + ":\t" + req.method + " " + req.url);
        if (req.url === '/hello') {
            res.writeHead(200);
            res.end('mTLS hello world\nfrom: ' + ownCert.subject.CN + '\nto client: ' + peerCert.subject.CN);
        } else {
            res.end();
        }
    }).listen(443);

Kafka​

const fs = require('fs')
const {Kafka} = require('kafkajs')

const kafka = new Kafka({
    brokers: ['kafka.kafka:9092'],
    ssl: {
        ca: [fs.readFileSync('/var/otterize/credentials/ca.pem', 'utf-8')],
        key: fs.readFileSync('/var/otterize/credentials/key.pem', 'utf-8'),
        cert: fs.readFileSync('/var/otterize/credentials/cert.pem', 'utf-8')
    },
})

const consumer = kafka.consumer({groupId: 'test-group'})

consumer.connect().then(
    consumer.subscribe({topic: 'mytopic', fromBeginning: true}).then(
        consumer.run({
            eachMessage: async ({
                                    topic,
                                    partition,
                                    message
                                }) => {
                console.log({
                    value: message.value.toString(),
                })
            },
        })
    )
)

Verify the generated mTLS credentials​

We can use openssl to inspect the generated certificates that were stored as Kubernetes secrets and mounted as volumes into pod containers.

To retrieve the credentials directly from the Kubernetes secrets store, fetch them via kubectl get secret:

kubectl get secret -n otterize-tutorial-mtls client-credentials-secret -o jsonpath='{.data.cert\.pem}' | base64 -d > svid.pem

Alternatively, retrieve the credentials from the container's mounted volume, e.g. by executing a shell command:

kubectl exec -n otterize-tutorial-mtls -it deploy/client -- cat /var/otterize/credentials/cert.pem > svid.pem

The retrieved credentials can be inspected with:

openssl x509 -in svid.pem -text | head -n 15

You should see a similar output to

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0b:eb:eb:4d:0e:02:7e:28:93:30:1c:55:26:22:8b:c7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = SPIRE
        Validity
            Not Before: Aug 24 12:19:57 2022 GMT
            Not After : Sep 23 12:20:07 2022 GMT
        Subject: C = US, O = SPIRE, CN = client.otterize-tutorial-mtls       # the client's name
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:

You can see that Otterize generated an X.509 keypair using the pod's name ("client") and namespace ("otterize-tutorial-mtls"): client.otterize-tutorial-mtls. The certificate belongs to a chain of trust rooted at the SPIRE server.

What's next​