Intro to Ray for AI on Kubernetes
This post finally continues my AI exploration series with a look at the open source solution Ray and how it can support AI workloads on Kubernetes.
What is Ray?
Initially created at UC Berkeley in 2018, Ray is an open-source unified compute framework to scale AI and Python workloads. The Ray project is predominantly managed and maintained by Anyscale.
At a high level, Ray is made up of 3 layers: Ray AI Libraries (Python), Ray Core, and Ray Clusters. These layers enable Ray to act as a solution that spans the gap between data scientists/machine learning engineers and infrastructure/platform engineers. Ray enables developers to run their AI applications more easily and efficiently across distributed hardware via a variety of libraries. Ray enables data scientists/machine learning (ML) engineers to focus on the applications and models they’re developing. And finally, Ray provides infrastructure/platform engineers with tools to more easily and effectively support the infrastructure needs of these highly performance-sensitive applications.
This diagram from the Ray documentation shows how the Ray Libraries relate to Ray Core, while Ray Clusters are a distributed computing platform that can be run on top of a variety of infrastructure configurations, generally in the cloud.
Through its various Python libraries in the Ray AI Libraries and Ray Core layers, Ray provides ML practitioners with tools that simplify the challenge of running highly performance-sensitive distributed machine learning-style applications on hardware accelerators. Ray Clusters are a distributed computing platform where worker nodes run user code as Ray tasks and actors. These worker nodes are managed by a head node which handles tasks like autoscaling the cluster and scheduling the workloads. Ray Cluster also provides a dashboard that gives a status of running jobs and services.
Ray on Kubernetes with KubeRay
Ray Clusters and Kubernetes clusters pair very well together. While Ray Clusters have been developed with a focus on enabling efficient distributed computing for hardware-intensive ML workloads, Kubernetes has a decade of experience in more generalized distributed computing. By running a Ray Cluster on Kubernetes, both Ray users and Kubernetes Administrators benefit from the smooth path from development to production that Ray’s Libraries combined with the Ray Cluster (running on Kubernetes) provide. KubeRay is an operator which enables you to run a Ray Cluster on a Kubernetes Cluster.
KubeRay adds 3 Custom Resource Definitions (CRDs) to provide Ray integration in Kubernetes. The RayCluster CRD enables the Kubernetes cluster to manage the lifecycle of its custom RayCluster objects, including managing RayCluster creation/deletion, autoscaling, and ensuring fault tolerance. The custom RayJob object enables the user to define Ray jobs (Ray Jobs, with a space!) and a submitter, which can either be directed to run the job on an existing RayCluster object, or to create a new RayCluster to be deleted upon that RayJob’s completion. The RayService custom resource encapsulates a multi-node Ray Cluster and a Serve application that runs on top of it into a single Kubernetes manifest.
When to Use Ray on Kubernetes
In many organizations, the people creating AI applications and the people running and managing the Kubernetes clusters and other infrastructure are different people, because the skills to do these types of activities well are specialized in nature.
As a platform engineer, you may want to consider encouraging use of Ray as a single scalable ML platform that members of your organization could use to simplify the development lifecycle of AI workloads. If AI practitioners are using Ray, you can use the KubeRay on Kubernetes to simplify onboarding and integration into your existing Kubernetes ecosystem. As a practitioner who is building AI applications, you may want to consider using or advocating for Ray if your organization is already using Kubernetes and you want to reuse the same code between development and production without modification, and to leverage Ray’s ML ecosystem with multiple integrations.
Ray Alternatives
Ray does not exist in a vacuum and there are numerous other tools that can support the same types of AI and Python workloads. Ray’s layered approach solves challenges from development to production that alternatives may not holistically address.. Understanding how Ray relates to other tools can help you understand the value it provides.
The Python Library components of Ray could be considered analogous to solutions like numpy, scipy, and pandas (which is most analogous to the Ray Data library specifically). As a framework and distributed computing solution, Ray could be used in place of a tool like Apache Spark or Python Dask. It’s also worthwhile to note that Ray Clusters can be used as a distributed computing solution within Kubernetes, as we’ve explored here, but Ray Clusters can also be created independent of Kubernetes.
To learn more about the relationships between Ray and alternatives, check out the “Ray & KubeRay, with Richard Liaw and Kai-Hsun Chen” episode of the Kubernetes Podcast from Google.
As in many other areas of tech, the right answer to the question “which tool is best for me” is, “it depends.” The best solution for your use case will depend on a variety of factors including which tools you’re comfortable with, how you divide work, what’s in use in your environment, and more.