{"id":91237,"date":"2020-06-10T11:00:40","date_gmt":"2020-06-10T18:00:40","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/security\/blog\/\/?p=91237"},"modified":"2023-05-15T23:06:36","modified_gmt":"2023-05-16T06:06:36","slug":"misconfigured-kubeflow-workloads-are-a-security-risk","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/security\/blog\/2020\/06\/10\/misconfigured-kubeflow-workloads-are-a-security-risk\/","title":{"rendered":"Misconfigured Kubeflow workloads are a security risk"},"content":{"rendered":"
Azure Security Center (ASC) monitors and defends thousands of Kubernetes clusters running on top of AKS. Azure Security Center regularly searches for and research for new attack vectors against Kubernetes workloads. We recently published a blog post about a large scale campaign<\/a> against Kubernetes clusters that abused exposed Kubernetes dashboards for deploying cryptocurrency miners.<\/p>\n In this blog, we\u2019ll reveal a new campaign that was observed recently by ASC that targets Kubeflow, a machine learning toolkit for Kubernetes. We observed that this attack effected on tens of Kubernetes clusters.<\/p>\n Kubeflow is an open-source project, started as a project for running TensorFlow jobs on Kubernetes. Kubeflow has grown and become a popular framework for running machine learning tasks in Kubernetes. Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs. This fact makes Kubernetes clusters that are used for ML tasks a perfect target for crypto mining campaigns, which was the aim of this attack.<\/p>\n During April, we observed deployment of a suspect image from a public repository on many different clusters. The image is ddsfdfsaadfs\/dfsdf:99<\/strong>. By inspecting the image\u2019s layers, we can see that this image runs an XMRIG miner:<\/p>\n <\/a><\/p>\n This repository contains several more images, which differ in the mining configuration. We saw some deployments of those images too.<\/p>\n Looking at the various clusters that the above image ran on showed that most of them run Kubeflow. This fact implies that the access vector in this attacker is the machine-learning framework.<\/p>\n The question is how can Kubeflow be used as an access vector for such an attack?<\/p>\n Kubeflow framework consists of many different services. Some of those services include: frameworks for training models, Katib and Jupyter notebook server, and more.<\/p>\n Kubeflow is a containerized service: the various tasks run as containers in the cluster. Therefore, if attackers somehow get access to Kubeflow, they have multiple ways to run their malicious image in the cluster.<\/p>\n The framework is divided into different namespaces, which are a collection of Kubeflow services. Those namespaces are translated into Kubernetes namespaces in which the resources are deployed.<\/p>\n In first access to Kubeflow, the user is prompted to create a namespace:<\/p>\n <\/a><\/p>\n In the picture above, we created a new namespace with the default name anonymous<\/strong>. This namespace is broadly seen in the attack and was one of the indicators to the access vector in this campaign.<\/p>\n Kubeflow creates multiple CRDs in the cluster which expose some functionality over the API server:<\/p>\n <\/a><\/p>\n In addition, Kubeflow exposes its UI functionality via a dashboard that is deployed in the cluster:<\/p>\n <\/a><\/p>\n The dashboard is exposed by Istio ingress gateway, which is by default accessible only internally. Therefore, users should use port-forward to access the dashboard (which tunnels the traffic via the Kubernetes API server).<\/p>\n In some cases, users modify the setting of the Istio Service to Load-Balancer which exposes the Service (istio-ingressgateway<\/strong> in the namespace istio-system<\/strong>) to the Internet. We believe that some users chose to do it for convenience: without this action, accessing to the dashboard requires tunneling through the Kubernetes API server and isn\u2019t direct. By exposing the Service to the Internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.<\/p>\n If attackers have access to the dashboard, they have multiple methods to deploy a backdoor container in the cluster. We will demonstrate two options:<\/p>\n <\/a><\/p>\n This image doesn\u2019t necessarily have to be a legitimate notebook image, thus attackers can run their own image using this feature.<\/p>\n <\/a><\/p>\n The Kubernetes threat matrix<\/a> that we recently published contains techniques that can be used by attackers to attack the Kubernetes cluster. A representation of this campaign in the matrix would look like:<\/p>\n <\/a><\/p>\n The attacker used an exposed dashboard (Kubeflow dashboard in this case) for gaining initial access<\/strong> to the cluster. The execution<\/strong> and persistence<\/strong> in the cluster were performed by a container that was deployed in the cluster. The attacker managed to move laterally<\/strong> and deploy the container using the mounted service account. Finally, the attacker impacted<\/strong> the cluster by running a cryptocurrency miner.<\/p>\n kubectl get pods –all-namespaces -o jsonpath=”{.items[*].spec.containers[*].image}”\u00a0 | grep -i ddsfdfsaadfs<\/em>\u00a0<\/u><\/strong><\/p>\n kubectl get service istio-ingressgateway -n istio-system<\/em><\/p>\n Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector: an exposed service to the internet. However, this is the first time that we have identified an attack that targets Kubeflow environments specifically.<\/p>\n When deploying a service like Kubeflow within a cluster it is crucial to be aware of security aspects such as:<\/p>\n To learn more about AKS Support in Azure Security Center, please see this documentation<\/a>.<\/p>\n\n
\n
How to check if your cluster is impacted?<\/h3>\n
\n
\n
Conclusion<\/h3>\n
\n