Using JetStack’s Kubernetes cert-manager to Automatically Renew TLS Certificates in the Ambassador API Gateway

Published in

Ambassador Labs

5 min readFeb 20, 2019

Security is of paramount importance for anyone exposing applications or services over the web. Therefore, the modern approach to threat modeling is highly valuable Although everyone needs to embrace security throughout the software design and build lifecycle, I want to focus on implementing effective Transport Layer Security (TLS) within Kubernetes, using the open source Ambassador API gateway and the JetStack cert-manager add-on.

Why is this Important?

TLS (and its predecessor SSL) are cryptographic protocols designed to provide secure communications over a network, primarily focusing on privacy and data integrity between a client and server. As part of this provision, there is a dependence on public key cryptography and the associated management of digital certificates.

To implement TLS on the web you need two things;

A gateway or proxy that provides TLS termination, i.e. an application that intercepts the encrypted communication, verifies the data, and decrypts the payload before handing this off to the upstream services
A tool to generate and manage the digital certificates used to verify public keys ownership.

If you deploy applications within Kubernetes, you can use the Ambassador Edge Stack API gateway to provide TLS termination and the cert-manager add-on to generate and manage the digital certificates.

TLS Termination and Generating Certificates Automatically within Kubernetes

Implementing TLS termination within Ambassador is thankfully trivial, and the documentation provides a walkthrough of the approach suitable for all skill levels. However, the tricky thing for many of us working at the operational front lines of websites and applications exposed over HTTP is ensuring that the TLS certificates can easily be generated for our domains, that they can be easily changed or “rotated” (particularly in the event of a breach), and that they are up to date. Ideally we want a certificate to be renewed before its expiration date automatically.

Cert-manager is a popular Kubernetes add-on from the good folks at JetStack, which automates the management and issuance of TLS certificates from various issuing sources. Although Ambassador has supported the use of cert-manager for quite some time, the latest 0.50.X release of the gateway includes a series of improvements, such as removing the need to restart the Ambassador process upon issuing a new certificate — everything is now automated.

Installation of Ambassador and cert-manager

Both the Ambassador API gateway and cert-manager have multiple installation options, but the simplest is typically to use the Helm Charts, e.g.:

$ helm install --name my-release stable/ambassador --version 1.1.0

$ helm install --name cert-manager --set webhook.enabled=false stable/cert-manager

I’ve made the assumption that you already have Helm and Tiller installed correctly, and for simplicity, I’ve chosen to not enable the ValidatingWebhookConfiguration, but you should understand the impact of this for production deployments.

Issuing Certificates: Notes on Challenge Types

Cert-manager issues certificates from a CA such as Let’s Encrypt. It does this using the ACME protocol, which supports various challenge mechanisms for verifying ownership of the domain..

Cert-manager currently supports two kinds of ACME challenges that enable domain ownership verification in different ways: HTTP-01 (validating over HTTP) and DNS-01 (validating over DNS).

It’s also worth noting that Wildcard domain certificates (those covering *.yourdomain.com) can only be requested using DNS validation. DNS Validation is also especially useful if the domains you are trying to get certification for are not public websites or cannot serve http requests on port 80.

There is information within the Ambassador documentation on how to implement both challenge types, and for the sake of brevity, I’ll only include the configuration here for the DNS-01 challenge.

Implementing a DNS-01 Challenge with cert-manager and Ambassador

The exact issuer configuration will depend on your DNS provider, but for this example, I’ll use AWS Route53, which I see a lot in the wild.

1. Create an IAM User with the IAM Policy specified in the AWS Route53 cert-manager documentation. Note the accessKeyID from the IAM user, and the current AWS region you are operating in, as you’ll need to replace the tokens in the YAML below with this data.

2. Create a Kubernetes secret named prod-route53-credentials-secret, which holds the secret-access-key with a value of the IAM User secret access key.

3. Create and apply a ClusterIssuer, replacing the “{}” tokens as appropriate:

---
    apiVersion: certmanager.k8s.io/v1alpha1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-prod
      namespace: default
    spec:
      acme:
        email: example@example.com
        server: https://acme-v02.api.letsencrypt.org/directory
        privateKeySecretRef:
          name: letsencrypt-prod
        dns01:
          providers:
          - name: route53
            route53:
              region: {REGION}
              accessKeyID: {SECRET_KEY}
              secretAccessKeySecretRef:
                name: prod-route53-credentials-secret
                key: secret-access-key

4. Create and apply a certificate:

---
    apiVersion: certmanager.k8s.io/v1alpha1
    kind: Certificate
    metadata:
      name: ambassador-certs
      namespace: default
    spec:
      secretName: ambassador-certs
      issuerRef:
        name: letsencrypt-prod
        kind: ClusterIssuer
      commonName: example.com
      dnsNames:
      - example.com
      acme:
        config:
        - dns01:
            provider: route53
          domains:
          - example.com

5. Verify the secret is created

$ kubectl get secrets
NAME                     TYPE                                  DATA      
ambassador-certs         kubernetes.io/tls                     2        
ambassador-token-846d5   kubernetes.io/service-account-token   3        
default-token-4l772      kubernetes.io/service-account-token   3

Now all that remains to do is test that you can access the service/website being secured over TLS (I personally still find curl the most useful approach: curl -v https://<your-domain> --output /dev/null).

With the configuration complete, cert-manager will ensure TLS certificates being served via Ambassador are valid and up to date periodically, and attempt to renew certificates at an appropriate time before expiry.

Wrapping Up

We all know the importance of making sure TLS certificates are renewed before they expire. The combination of Ambassador 0.50 and cert-manager makes this super easy by automating much of the process.

I hope the above guide has been useful, but I’m always happy to answer questions, and the Ambassador Labs community is also keen to receive pull requests against our documentation if you want to make a correction or clarification.

Visit the quick start guide to get started with Ambassador Labs today. You can find more information about configuring cert-manager with Ambassador in the documentation, and you can ask any questions on our Slack Community or raise issues via GitHub.