Several times recently on the Kubernetes Podcast from Google, which I co-host alongside my colleague Abdel Sghiouar, the concept of “in-tree” vs “out-of-tree” in upstream Kubernetes has come up. I think it’s an interesting topic, so I thought I’d write a little post to explain it. (If you want an intro to Kubernetes, you can check out mine here!)
What is the “Tree”?
In the last couple of Kubernetes release lead interviews, for 1.28 and 1.29, some of the major changes in the releases have been the movement of certain functionalities “out-of-tree.” The Kubernetes “Tree” is, simply, the Kubernetes codebase. It makes sense to refer to it as a tree because of code organization and management. Code “branches” and the structure of the source controlled change history when represented visually, are a couple of aspects that make a codebase “tree-like.” And that’s what’s being referred to here.
Functionality moving out of the Kubernetes “tree” is an important trend with major implications for the community and the Kubernetes project itself. There was once a time when Kubernetes’ functionality wasn’t “simple,” but it was much simpler than it is now. The early days, when the project, the community, and the codebase, were much more manageable. At that time, when folks wanted to add some functionality to Kubernetes, there was room to do it. And so that’s what people did.
The Storage Plugin Problem
For example, storage providers made their own plugins so people could use their storage solutions with Kubernetes, and they put the code for these plugins right into the Kubernetes code base. The great thing about this approach is that for anyone who uses Kubernetes, if they want to use any kind of storage that has an in-tree plugin, they don’t have to do anything extra, the functionality is already there! That makes for a great user experience, right?
The problem with this is that as the project has grown, more and more people, code, and tools all have to work together. This makes releases more and more challenging for everyone. Imagine, if someone from a specific company had to maintain that company’s storage driver in Kubernetes, they couldn’t just make an update to it when the company’s api or product changed, even if it changed significantly. They would have to wait and work with the rest of the project to get any changes into the next Kubernetes release. If the maintainer of the plugin didn’t contribute elsewhere in Kubernetes, they might not be very familiar with Kubernetes’ release processes.
Ultimately, there’s a lot of potential for delays and bad user experiences introduced by embedding vendor-specific code into Kubernetes itself. Not to mention the general complexity and bloat it adds to Kubernetes, which at this point, is already a giant, complex code base. The storage plugin issue is such a big project, there’s even a blog on kubernetes.io about the ongoing effort to move these storage plugins out-of-tree.
The “Out-of-Tree” Trend
This concept of moving functionality out-of-tree to manage complexity and release processes has been so well-received, and the Kubernetes codebase has grown so enormously complex, that the concept has gained traction in use cases beyond storage plugins as well. The new Gateway API is also an extension of this concept.
The Gateway API is an exciting project meant to fix some questionable design decisions from the early days of Kubernetes. The original Kubernetes Ingress API was designed to solve a problem that maintainers thought users would eventually have. But it was designed so early on, that very few users actually needed to manage ingress at the time. Thus, the original implementation of Ingress in Kubernetes involved a lot of theory and guesses. Now, Kubernetes has many users in production with very specific ingress needs. The community has learned that a lot of these early guesses were wrong in ways that aren’t easy to fix in-place. The Gateway API is a reimagining of ingress management functionality in Kubernetes based on real-world use cases.
One consideration that makes ingress in Kubernetes hard is that there are many different environments a cluster could exist in. Each environment might have its own tools and unique ingress traffic considerations. Since there are many tools and environments users might use, it’s kind of a similar problem space to that of the storage drivers. The best ingress solution for a cloud environment could be provided by the cloud provider, for example. So the Gateway API, rather than creating a new version of the Ingress API in-tree, created a new standard that vendors could use to create their own implementations out-of-tree.
Your “Tree” of Knowledge
The simple term “out-of-tree” hides a lot of context and fascinating history of the Kubernetes project. As an open source project, the number of environments and tools it can interact with is practically limitless. Containing the complexity of those interactions is very important to the continued health and growth of Kubernetes. Vendor solutions for storage, networking, and much more, have an important role to play in the ecosystems around Kubernetes. By keeping code for this multitude of possibilities just outside of Kubernetes itself, we aim to keep the project more manageable. While Kubernetes is complex, hopefully this history lesson has helped you understand it just a little better.