Deploying Our Pull Requests in Flight: Previewing WIP in K8s

Testing Our Running Code Changes During Review

Published in

Ambassador Labs

5 min readApr 27, 2022

At Ambassador Labs, we’ve been using Telepresence for our local development of the Ambassador Cloud platform. We were aware Telepresence was used by individual developers to access our staging and production APIs in order to iterate quickly on code changes against a remote environment.

At the beginning: Single-threaded review loops

Over time, we actually saw tremendous adoption of Telepresence by our frontend developers, as it enabled them to run the UI locally while interacting with our distributed APIs easily and securely in shared remote Kubernetes environments. Using Telepresence for UI development proved to be very efficient, allowing for fast iterations, “hot reloads” and the usage of IDE debugging tools — this is especially true when compared to the alternative inner dev-loop of building, pushing and running Docker containers.

We also saw fewer and fewer developers relying on mock data and locally-hosted backend services. However, we realized our Telepresence workflow fell short when it came to sharing still-in-development previews of changes with teammates, whether they are other developers, UX designers, product people or marketing folks.

The value of long(er) running preview environments

As a distributed company, we don’t expect reviews to happen immediately. Yet, as engineers, we know how the feedback loop is a critical element to our project’s success. The reality is that developers can only work (and run) on one changeset at a time (i.e. one branch). We asked ourselves this question:

How can we make our developers’ workflow more efficient by not having them blocked on feedback for work ready for review?

We partly knew the answer to this question since we were already familiar with Netlify and their deployment preview functionality for our statically-hosted corporate and documentation website. In a nutshell, every pull request gets built and deployed with a unique URL available to preview changes before merging them. This turns out to be much more efficient than looking at Markdown changes and screenshots.

Telepresence in CI a.k.a Deployment Previews

Instead of running our changes locally, we looked at ways to integrate Telepresence in our pull request and CI workflow. Our objective was that whenever developers would push code changes to a GitHub pull request, we would automatically deploy their application through status checks on a shared Kubernetes environment.

After building the pull request and pushing a Docker image containing the changes, we generate a Kubernetes manifest using Kustomize that duplicates the original application Deployment resource while injecting the Telepresence pod-daemon. This pod-daemon sidecar and Ambassador Cloud will then work together to generate and host a unique preview URL for every pull request deployment. At a high level, this is what our CI workflow looks like:

We’ve documented this setup in details on our website.

As with regular Telepresence intercepts, Ambassador Cloud will proxy traffic destined to preview URLs and tunnel requests inside our Kubernetes cluster, letting the Telepresence traffic-agent reroute requests destined to our pull request Deployment without having to modify our ingress-controller or API gateway configuration. The difference here is that this deployment preview URL is long-lived (or at least longer-lived than we expect our developers’ laptop to be online with the running version of the changes under review) and allows anybody in the organization to test the changes in isolation.

Reviewers no longer only look at code changes. They also look at the UX changes, making code reviews more efficient. Our preview URLs are secure, authorized, restricted to team members and easy to share.

Authentication protects our preview URLs from being accessed by people outside the organization — this is built-in to Ambassador Cloud using state-of-the-art OAuth SSO. But since we are developing our Ambassador Cloud product (which requires authentication) on a platform that enforces its own authentication mechanism, we had to augment our authentication flow. For this reason, we decided to equip all of our preview URLs with additional request headers that inject user tokens into requests going through our preview URL. This will also work great in your team if you are using a different auth scheme like Basic access authentication as well.

Fast feedback, less FOMO, and enhanced dev productivity

Integrating automated deployments for every pull request and attaching a preview URL to intercept deployments to a specific version allowed us to be more effective as a team, improving our review process, working asynchronously, and getting the changes in front of more stakeholders before shipping to production.

“As a marketer, deployment previews let me provide more actionable feedback to the engineers and get my projects shipped to customers faster. Before deployment previews I requested changes from screenshots of staging environments, but I was missing context. Now I can click through the new version of our app before it goes to production and request better changes.”
- Kelsey Evans, Growth Marketer

Of course, tearing down our preview deployment is super easy. Since we integrated all our deployment steps with ArgoCD, we can execute actions whenever pull requests are closed or merged to cleanup (purge) the target deployment for a specific pull request, thus terminating the preview.

Get started today with Telepresence and enjoy fast local development along with deployment previews for your team.