CONFERENCE SUMMARY

KubeCon NA 2021 Key Takeaways: DevX, Security, and Community

It’s that time of year again when we get to reflect on another awesome KubeCon NA. Although I couldn’t attend the event in person this year, several (US-based) members of the Ambassador Labs team made the journey to Los Angeles and I followed along with the virtual component of the event.

Firstly, I’ve got to give massive kudos to all of the organizers at the CNCF and in the wider community — I know from past experience of running (smaller) hybrid events that it’s actually more work than running two individual conferences! And speaking of other events, we also ran our inaugural Dev House virtual conference (a successor to Ambassador Fest!) alongside the KubeCon days but during European/African hours, so I’ll share highlights from the talks here, too.

Perhaps no surprises here, but my key takeaways from KubeCon NA 2021 build upon the core observations from KubeCon EU earlier in the year:

• Developer experience (DevX/devex) is now a top priority for vendors, open-source projects, and platform teams
• Decision criteria for platform build vs buy will get more attention over the next year
• Cloud native security is getting the focus and solutions the space deserves
• Multi-cloud is a thing (again!), and this time it’s all about Kubernetes
• (Enterprise) adoption of multi-cluster is driving conversations and practical solutions in this space; networking is top of mind
• Anyone can contribute to the community, but take care of yourself and others

And if I had to pick a general cross-cutting theme for the event, it would be “maturity”. The cloud native ecosystem appears to have “crossed the chasm” of being accepted within the traditionally more technologically conservative enterprise landscape. The event mirrored this with a stronger focus on use cases from enterprise end-users, exploration of practical security considerations, and discussions on how to build cloud platforms and software at scale. Katie Gamanji framed it perfectly in her opening keynote:

Developer experience is now a top priority for vendors, open-source projects, and platform teams

Although several of the Ambassador Labs team kicked off the week by presenting and attending at EnvoyCon (which looked great!), I was presenting a session at the DevX Day colocated event. And here the topic of developer experience was front and center. Shout out to Pauline Narvas and the entire DevX Day program committee for assembling a great lineup of topics.

Key themes that will be weaved throughout the blog post showed up early at DevX Day, with Rin Oliver presenting a great talk on the value of security within open source communities. The core themes of this talk were making security a first-class concern and making it easy to do the right (secure) thing within a community.

I also learned about the “kui” kubectl augmenting/replacing tool located in the K8s SIGs GitHub repo. The tagline is “Kui takes your normal kubectl command line requests and responds with graphics.” You can clearly see a lot of thought about devex has gone into this tool:

There was a great case study on the impact of devex by Patricia Gaughan and Ana-Maria Cālin from InfluxData, “From Villains to Heroes: How an Improved DX Has Made Our Devs Happy-ish”. Lots of focus here was on creating an effective “remocal” (remote-local) development environment/experience for their engineers to code and debug services running in Kubernetes. As a contributor to Telepresence, it was great to see how influential this CNCF tool had been:

The Okteto team was up next with Jacob MacElroy talking about the value of preview environments. This is something that is getting quite a bit of attention at the moment (e.g. Kostis Kapelonis’ great post “Unlimited Preview Environments with Kubernetes Namespaces”), and we’re also seeing success with creating previewURLs for personal intercepts/development via Telepresence. Ramiro Berrelleza presented a thought-provoking session on the divergence of development and production environments over recent years. Expect lots of interesting discussions to emerge around this topic, and Ramiro hinted that there could be opportunities to create useful standards in this space (perhaps a custom resource to describe a development environment?)

Next, I presented my session “From Kubernetes to PaaS to Developer Control Planes”, focusing on what is needed for a good developer experience when interacting with software delivery tooling and platforms. My core advice for folks building platforms was:

• Treat platform as a product
• Realize that you can’t have good DevX without good UX
• Focus on workflows and tooling interoperability (“developer control planes”)

The slides can be found here, and please reach out with questions:

On a related topic, Max Körbächer from Liquid Reply presented “A Cloud Native Foundation For Developer/Application Platforms” and shared some great lessons from building what he referred to as “internal development platforms (IDPs).”

I’ve seen the Humanitec folks talk about the same concept of IDPs and they have also created the interesting resource that I have made a (small) contribution to: https://internaldeveloperplatform.org/

In general, I thought the DevX Day event was excellent and covered a timely topic. I would be keen to see this at future KubeCons. Hat tip to Ramiro for his live-tweeting skills through the event; he created a single thread with all of his DevX Day takeaways. And lots of great discussions also happened in the associated CNCF Slack channel, and I want to say a big thank you to everyone for sharing their devex stories.

Developer experience was also highly visible in the KubeCon keynotes, with Jasmine James from Twitter delivering an outstanding talk on “human-centered developer experience”. I live-tweeted the talk in the thread here:

As Jasmine was setting the context by introducing Janis, a machine learning engineer struggling with her day-to-day cloud native development work due to existing and new tooling being rolled out at the new company, I could almost feel the collective empathy in the audience through my screen!

Decision criteria for platform build vs buy will get more attention over the next year

There were several outstanding talks on building platforms at KubeCon. In the Thursday keynote, Robert Duffy, Vice President of Development and Runtime Platform at Expedia presented “Building Support For Your Cloud Native Journey”. This was the story of how the Expedia Group Common Developer Platform was built. Duffy was super clear that a platform cannot be built by engineering alone: “To build a platform you need engineering support, product management, and executive buy-in” and continued by sharing his experiences of how successful (and unsuccessful) teams approach platform re-build projects. This was pure gold:

I was also very impressed with the focus on (developer experience) metrics related to the platform, and the goals they were aiming for.

Another outstanding platform talk that focused even more on the effective inner/outer developer loops was Improving Dev Experience: How We Built a Cloud Native Dev Stack At Scale by Srinidhi S and Venkatesan Vaidyanathan from Razorpay. They began by explaining the pain points of cloud native development using Kubernetes, such as the slow feedback of the container build-push-deploy-test loop, and ran through several of the solution iterations they tried. This included Telepresence (network-based proxying between local machine and remote cluster), DevSpace (file-based code syncing), CompileDaemon (for auto-reload), Helmfile (for managing service dependencies), and LocalStack (for mocking AWS dependencies).

I also enjoyed “How Salesforce built an extensible PaaS using CRDs saving devs 4200 hours” by Mayank Kumar, and live-tweeted this talk in a thread:

There was some great discussion in this talk about using Kubernetes Custom Resources to define applications and supporting infrastructure, which also reminded me of the great work going on from both the Crossplane and Open Application Model (OAM) communities. There is definitely a lean into “everything as [declarative] code” within the community.

As I was watching all of these talks, I couldn’t help but think back to Kelsey Hightower’s evergreen tweet from 2017:

And by mentioning this I’m not attempting to criticize the presenters or organizations in any way, as the talks were superb. However, I couldn’t help feeling that this kind of presentation only encourages other folks (perhaps not at the same scale) to start building out their own platform, when really something more opinionated or “off the shelf” would do. Perhaps at the next KubeCon it would be good to hear more adoption stories of Heroku, CloudFoundry, OpenShift, or other PaaSs?

Personally, I believe that this PaaS-like space is an area where a lot of change will happen in the next year or so. At Ambassador Labs we are placing our bets on an opinionated but flexible layer that is being referred to as a developer control plane.

The goals are to help developers code, ship, and run applications without writing lots of YAML, while still building on CNCF technologies, such as Argo CD, Argo Rollouts, Emissary-ingress, etc. If you want to know more, check out our Kubernetes in 5 challenge, where you get hands-on with coding, shipping, and running an application in K8s in under 5 minutes.

At our Dev House event Dave Sudia, CTO at UPchieve shared his experiences of using a developer control plane to enable the fast, and cost-effective, development and delivery of software using Kubernetes.

On a related topic, I thoroughly enjoyed the “Shifting Spotify Engineering from Spreadsheets to Backstage” KubeCon talk by Johan Haals and Patrik Oldsberg. This provided insight into the journey from using a spreadsheet to track service ownership and status all the way through to the development and use of the Backstage service catalog project (which was recently donated to the CNCF).

I live-tweeted the talk here:

The entire thread is worth reading, as there were a lot of good takeaways, but in particular I liked the advice on how to encourage the adoption of a service catalog in your organization: “Try the carrot, rather than the stick!” and focus on:

• Creating incentives for developers to add their service to the catalog
• Start small: look for organizational pain points or teams struggling with lack of service/dependency visibility
• Prioritize efforts

Expect lots more discussion on platform build vs buy over the coming year!

Cloud native security is getting the focus and solutions the space deserves

It was good to see the importance of security called out at several points in the keynotes, and for me, this is definitely a sign of maturity within the ecosystem. I thoroughly enjoyed Frederick Kautz and Allan Friedman’s “SBOM is Coming: Why You Should Care and How You Can Help”. With the recent focus on software supply chain issues, this was a timely talk.

As I mentioned on the Twitter thread, if you’re interested in SBOMs and the cloud native buildpacks project then check out Ram Iyengar’s recent InfoQ article “Using Cloud Native Buildpacks to Address Security Requirements for the Software Supply Chain”

At Dev House we had Kostis Kapelonis talk about the potentially tricky subject of managing secrets when using a GitOps approach to continuous delivery. Kostis did a fantastic deep dive into using Bitnami’s Sealed Secrets for K8s:

As honorable mentions in the cloud native security space, there was also a lot of discussion on:

• SPIFFE as an identity control plane
• OpenSSF as a vendor neutral organization to improve security through beginning together with tech leaders, online communities, and creating best practices
• The Falco project for implementing cloud native runtime security

Multi-cloud is a thing (again!), and this time it’s all about Kubernetes

Even without the presence of VMware and Google in the sponsor showcase, there were still a lot of vendors talking about the topic of multi-cloud, e.g., D2iQ, Tetrate, TriggerMesh, and Buoyant to name a few. Also running at the same time as KubeCon was the Google Cloud Next event, and this great summary article from El’ Reg captures how much of a focus multi-cloud is for Google now.

A repeated theme I heard at KubeCon was the importance of adopting GitOps practices when adopting multi-cloud or multi-cluster environments. You can probably manage a couple of Kubernetes clusters through scripting or manual intervention, but as we’ve seen with VM management in the past, once you get past a couple of instances the power of automation becomes obvious.

The colocated GitOpsCon event contained many great stories and use cases from large organizations that testified to this. For example, Cisco’s adoption of a cellular architecture for multicluster:

On a related note, it was great to see that the Kubernetes Cluster API has recently reached production readiness with version 1.0.

The New Stack pancake breakfast provided great thinking points for folks looking to adopt multi-cloud, and a key message was focusing on standardizing the workflows rather than the tech:

And finally, not a KubeCon thing per se, but I saw this related article from Cloud Pundit floating around in Twitter threads last week: “Multicloud failover is almost always a terrible idea”. This is well worth a read!

(Enterprise) adoption of multi-cluster is driving conversations and practical solutions in this space; networking is top of mind

Kaslin Fields presented a great multicluster-themed call-to-action in the Wednesday keynote “From One to Many, the Road to Multicluster”. She began by highlighting the use cases of multicluster, including geographical/hybrid environments, billing management, and security and compliance. If you have been using public cloud technologies at scale for some time, all of these use cases will ring true; I saw many examples of this during my time as a cloud engineering consultant in London a few years ago.

Kaslin highlighted that the most challenging component of multicluster is often the networking. Historically there hasn’t been great support here in the world of Kubernetes, although I wanted to shout out SIG-Multicluster for the Cluster API and also the Linkerd team for their multicluster service mirroring implementation (Thomas Rampelberg and I presented a KubeCon session about this in the past).

The new Gateway API and Multicluster Ingress work is super interesting (and we’ve been working on supporting this in Emissary-ingress and the Ambassador Edge Stack) and the (Google sponsored?) Multicluster Services (MCS) that Kaslin mentioned is definitely worth a look — at first glance, this does look similar to the Linkerd service mirroring concept mentioned above.

At Dev House, Emissary-ingress lead engineer Flynn shared his experiences with dealing with the challenge of K8s ingress and networking. One of his primary calls-to-action was the importance of good documentation for projects in this space — and not just technical docs, the getting-started (and general developer experience focused docs) are super important:

We also had a fantastic talk from Krishna Modi from PharmEasy, who talked about the challenge of scaling ingress when dealing with a massive usage spike, and he also discussed why they hadn’t yet implemented multi-region ingress (although they do run their applications from multiple availability zones):

A related recent case study about implementing multi-region ingress in K8s can be found from the Monday.com team: “monday.com’s Multi-Regional Architecture: A Deep Dive”

Anyone can (and should) contribute to the community, but take care of yourself and others

The Friday keynote contained some fantastic calls to action to get involved with the cloud native community, from Christoph Blecker and Paris Pittman’s “Sustaining a Contributor Community’s Next Generation” to Frederick Kautz and Allan Friedman’s “SBOM is Coming: Why You Should Care and How You Can Help”. However, the stand-out presentation for me was from Stephen Augustus, “Maintaining the Maintainers”, and his focus was on being kind and not overburdening yourself by continually taking on more responsibility — especially given the strange state of the world over the past two years.

It was also great to see members of the CNCF community acknowledged in the annual awards section. Congratulations to Anais Urlichs and all the award winners:

At our Dev House event many of our Ambassador Community Advocates “rocked the mic”, and shared their knowledge in panels and presentations. Special thanks to Daman Arora and Prayag Savsani for their continued awesome support!

I’ve also got to shout out Cheryl Hung, Mario Loria, Kasper Nissen, and Damian Marquez for their contributions to our recent community Ambassador Labs podcast series, which my awesome colleague Edidiong shared in her tweets of my Dev House talk:

And, Erika, another of my awesome colleagues, explored these ideas more in more depth in her Dev House talk:

Check out the related The New Stack article that draws on ideas, too: “How the Developer Experience Is Changing with Cloud Native.”

Miscellaneous observations

Any KubeCon summary is obviously written with a bias on the author’s interest, and this piece is no different. Although I wasn’t actively focused on some of these topics or technologies, I wanted to highlight the themes I repeatedly observed:

eBPF was everywhere, from Cilium for communications to Falco for security scanning. Liz Rice’s talk on this topic was very popular.

Open Policy Agent (OPA) received a lot of attention, particularly around Gatekeeper admission controller, and the Styra folks were leading the way here.

We saw a lot of love for the CNCF Telepresence project (that the Ambassador Labs team originally created). Hat tip to Peter ONeill for presenting a great Telepresence talk at KubeCon, “From Network Engineer to K8s Developer: Lessons Learned via Telepresence”, and Kaslin Fields did a fantastic job live-tweeting this:

At Dev House we had Ryan Ross present a great use case of Telepresence for operations at Martin and also Jose Cortes presented a deep dive into the new TUN device that has been implemented in Telepresence:

There was a lot of K8s and cloud 101 content, and this was very popular. Our very own Alice Wasko presented “Homebrewing a Kubernetes Bootcamp: From College to K8s Support Engineer”, and we ran two oversubscribed workshops at Dev House, focusing on K8s 101 content and getting hands-on with Argo CD and Argo Rollouts.

We’ll be running more of these in the future, and so please reach out if you want to get involved. In the meantime, check out our Kubernetes Developer Learning Center, which is the “class textbook” for our workshops.

Wrapping up!

And that’s a wrap for KubeCon NA 2021. Many thanks to all of the presenters, attendees, and organizers at KubeCon! And a big thanks also to everyone we chatted to at KubeCon, whether it was via the booth, Slack or at our Dev House event.

Please stay in touch, and we look forward to seeing you in Valencia (hopefully in person!) at KubeCon EU 2022!