What’s new

Mitigations methods

Understanding the attack surface of containerized environments is the first step of building security solutions for these environments. In addition to helping organizations measure and assess coverage of threats with matching detections, the updated threat matrix for Kubernetes can now also help organizations with a systematic approach to apply mitigation techniques that prevent attacks from being successfully launched.

In this third version of the threat matrix, we introduce a collection of mitigations specific to Kubernetes environments and associate each with relevant threat techniques. Those mitigations, as displayed below in Figure 1, provide practical tools to prevent the various attack techniques, using built-in Kubernetes and cloud tools.

When reviewing the different threat techniques in the matrix, a list of relevant mitigations is provided so that organizations can see if they are taking all the necessary steps to prevent a threat. Additionally, when looking at a specific mitigation, a list of relevant threat techniques is displayed and can help organizations prioritize their mitigation implementation plan according to their threat assessment and detection coverage in each area.

Mapping to MITRE ATT&CK techniques

Last year, MITRE added a container matrix to the MITRE ATT&CK framework. MITRE ATT&CK for containers matrix, inspired by Microsoft threat matrix for Kubernetes, is a result of a joint effort between MITRE, Microsoft, and additional companies in the industry. The differences between Microsoft’s and MITRE’s matrices are described in this blog. In the new version of the Microsoft threat matrix for Kubernetes, we include a mapping between the Microsoft matrix and MITRE ATT&CK techniques and mitigations, as displayed below in Figure 2. This can help organizations to efficiently use the two frameworks.

MITRE ATT&CK matrix for containers does not have an equivalent technique for each of the techniques in the Microsoft threat matrix for Kubernetes. When there is no equivalent technique in the MITRE matrix, the Microsoft techniques might be mapped to a MITRE technique that is not part of MITRE’s containers matrix but shares the same principle. For example, the Backdoor Container (MS-TA9012) technique explains that attackers can use Kubernetes controllers (such as daemonsets) to run their code and survive reboots of the pods\nodes. This is very similar to MITRE’s Create or Modify System Process technique (T1543), which is about using services\daemons for the exact same purpose. Another example is the mapping between Malicious Admission Controller (MS-TA9015) and MITRE’s Event Triggered Execution. Although MITRE doesn’t talk about containerized environment, those two techniques share the same idea. In cases when there is no matching MITRE technique with the same principle, the Microsoft technique will not point back to a MITRE technique.

New techniques

The new version of the matrix also introduces two new techniques and additional re-categorization of existing techniques:

1. New technique: Static pods

A persistence technique which allows attackers to deploy pods that aren’t managed by the Kubernetes API server.

2. New technique: Collecting data from pod

Kubernetes-native technique which allows attackers to extract data from running pods.

3. Extending existing technique: Container service account

Attackers may create new service accounts or steal tokens of existing service accounts for future use from inside and outside the cluster. Therefore, we also added this technique to the persistence tactic.

4. Extending existing technique: Exposed sensitive interfaces

Attackers may use management interfaces for discovery purposes, after gaining initial access to the cluster. By using the network reachability between pods, attackers can connect to management interfaces from the internal network, allowing them to get valuable information about the workload. Thus, we also added this technique to the discovery tactic.

New web interface

As new threats were added to the Kubernetes matrix and additional content was introduced, it became increasingly harder to effectively deliver the breadth of information included in the matrix as a blog post. Looking for ways to make it easier to use the Kubernetes threats and mitigations matrix as reference material for day-to-day security operations, we are releasing the matrix as a web site, shown in Figure 4 below.

The new version of the threat matrix for Kubernetes is now available at: http://aka.ms/KubernetesThreatMatrix

The threat matrix for Kubernetes can help organizations to have visibility to the unique attack surface of Kubernetes and help them to measure their coverage to those threats. With the new mitigation section, organizations can now understand the measures required to prevent those threats.

Microsoft Defender for Cloud can help detect and mitigate threats in your Kubernetes environments. Learn more about Microsoft Defender for Cloud support for container security.

The post Mitigate threats with the new threat matrix for Kubernetes appeared first on Microsoft Security Blog.

As containers become a major part of many organizations’ IT workloads, it becomes crucial to consider the unique security threats that target such environments when building security solutions. The first step in this process is understanding the relevant attack landscape.

The MITRE ATT&CK® team has received frequent questions from the community about if or when ATT&CK would include coverage for adversary behavior in containers. Previous iterations of ATT&CK have included references to containers (for example, Resource Hijacking) and some clearly container-relevant techniques (for example, Implant Internal Image), but the coverage was insufficient to provide network defenders a holistic view of how containers are being targeted in enterprise environments.

Addressing the need for a common framework for understanding container threats

Given clear community interest, inspiration from Microsoft’s work on the threat matrix for Kubernetes, and the publication of research from other teams, the Center for Threat-Informed Defense launched an investigation (sponsored by several Center members including Microsoft) that examined the viability of adding containers content to ATT&CK. The purpose of the Container Techniques project was to investigate adversarial behavior in containerization technologies and determine whether there was enough open-source intelligence to warrant the creation of an ATT&CK for Containers matrix, resulting in either new ATT&CK content or a report on the state of in-the-wild Container-based tactics, techniques, and procedures (TTPs). The Center’s research team quickly concluded that there was more than enough open-source intelligence to justify technique development, ultimately resulting in the new matrix.

As of the ATT&CK v9 release, the ATT&CK for Containers matrix is officially available. More details about the Containers matrix can be found in MITRE-Engenuity’s announcement blog. Some highlights of the new matrix include related software entries, procedure examples to help network defenders better understand new container-centric techniques, data sources to match the recent ATT&CK data sources refactor, and many others.

Figure 1. ATT&CK for Containers matrix.

Evolving the threat matrix

MITRE ATT&CK has become the common vocabulary for describing real-world adversary behavior. ATT&CK offers organizations a method to measure their defenses against threats that impact their environment and identify possible gaps. With ATT&CK’s approach of methodically outlining the possible threats, Microsoft built the threat matrix for Kubernetes, which was one of the first attempts to systematically map the attack surface of Kubernetes. An updated version of the matrix was released earlier in 2021.

Figure 2: Threat matrix for Kubernetes.

Microsoft took part in the Center’s project and contributed knowledge that the company gained in the field of container security. Microsoft’s unparalleled visibility into threats helps to identify real-world attacks against containerized workloads and provide information about tactics and techniques used in those attacks. One example of such an attack is a cryptocurrency mining campaign that targeted Kubernetes. In this incident, Microsoft saw evidence of the following techniques from the Microsoft threat matrix:

• Exposed sensitive interfaces
• New container
• Pod/container name similarity
• List Kubernetes secrets
• Access Kubernetes API server
• Resource Hijacking

The techniques that went into ATT&CK for Containers are different from those in the Microsoft threat matrix. As described in a blog post by the Center, it was preferable to use an existing ATT&CK technique rather than create a new one when possible. Therefore, several techniques from the threat matrix were mapped into existing Enterprise ATT&CK techniques. For example, in the techniques listed above, “Exposed sensitive interfaces” from the threat matrix is equivalent to ATT&CK’s “External Remote Services.”

The Center’s process for leveraging Microsoft’s Kubernetes threat matrix was as follows:

• Cross-referencing threat intelligence with the techniques in the Kubernetes threat matrix.
• Determining whether techniques with sufficient intelligence backing were already covered by existing Enterprise ATT&CK techniques, or whether they justified the creation of one or more new techniques or sub-techniques.

Considering Microsoft’s tactics mapping for specific techniques and how they fit within ATT&CK’s Enterprise, Cloud, and Containers matrix scoping, as in the case of multiple forms of “lateral movement,” the Center instead identified pivots from one ATT&CK platform matrix to another (for example, Containers to Cloud).

The following are examples of techniques from Microsoft’s matrix that were re-scoped to fit into existing Enterprise ATT&CK techniques:

Meanwhile, the following are examples of techniques from the Microsoft threat matrix that were re-scoped based on the Center’s platform decisions and additional open-source intelligence, with additional detail on each technique/sub-technique available in its description within ATT&CK for Containers:

Not all the techniques and tactics that appear in the Microsoft threat matrix went into the new ATT&CK matrix. ATT&CK focuses on real-world techniques that are seen in the wild. In contrast, many of the techniques in the threat matrix were observed during research work and not necessarily as part of an active attack. For example, “CoreDNS poisoning” from the updated matrix is a possible attack vector but hasn’t been seen in the wild yet.

ATT&CK is dynamic

ATT&CK for Containers is by no means finished, and we look forward to future additions based on new intelligence and further community contributions. Before the public release of ATT&CK for Containers, Microsoft released an updated version of the threat matrix for Kubernetes, which speaks to the fast-paced evolution of this technology space and the need to keep up with new adversary behaviors.

The next step for the ATT&CK team is to assess the new content in Microsoft’s matrix and consider it for potential future inclusion in ATT&CK based on the factors described above. Microsoft and the ATT&CK team will continue to collaborate to ensure that container techniques coverage in ATT&CK is up-to-date and can continue to serve the need of the community.

With the completion of this Center project, ATT&CK for Containers will be maintained by the ATT&CK team, who would love your continuous feedback and contribution! Let the team know what you think, what could be improved, and most importantly what you see adversaries doing in the wild related to containers. Feel free to send an email at any time to attack@mitre.org. If you have ideas for other research and development projects that the Center should consider, please send an email to ctid@mitre-engenuity.org.

Learn more

To learn how Microsoft can help you protect containers and relevant technologies today, read about Microsoft Defender for Endpoint and Azure Defender.

To learn more about Microsoft Security solutions, visit our website. Bookmark the Security blog to keep up with our expert coverage on security matters. Also, follow us at @MSFTSecurity for the latest news and updates on cybersecurity.

The post The evolution of a matrix: How ATT&CK for Containers was built appeared first on Microsoft Security Blog.

Since the publication of the threat matrix last year, things have changed:

• New threats were discovered as attackers targeted more and more Kubernetes workloads.
• We were glad to see that the security community adopted the matrix and added more techniques.
• As Kubernetes evolves, it becomes more secure by default and some techniques are no longer relevant.

Today, we are releasing the second version of the threat matrix for Kubernetes, which considers these changes. The updated matrix adds new techniques that were found by Microsoft researchers, as well as techniques that were suggested by the community. We also deprecate several techniques, which do not apply anymore to newer versions of Kubernetes. In this version, we also add a new tactic taken from MITRE ATT&CK®: collection.

What has deprecated?

Kubernetes evolved and became more secure by default; techniques that appeared in last year’s matrix aren’t relevant to newer environments. Therefore, we decided to deprecate some of the techniques:

Exposed Kubernetes Dashboard

The Kubernetes dashboard’s usage has been in decline for a while now. Cloud-managed clusters, such as Microsoft’s AKS and Google’s GKE, deprecated this service and moved to a centralized interface in their portals. Moreover, the recent versions of the Kubernetes dashboard require authentication and it’s less likely to find exposed dashboards that do not require authentication. Consequently, we also removed the technique, “Access Kubernetes dashboard,” under lateral movement tactic. Older versions of Kubernetes, including newer clusters that have the dashboard manually installed, are still affected by this technique. The concept of this technique was generalized in the new technique: exposed sensitive interfaces (see below).

Access tiller endpoint­

As of version 3, Helm doesn’t use its server-side component, Tiller. This is a major improvement in the security of Helm. Currently, Helm operates (by default) on behalf of the user’s credentials as they appear in the kubeconfig file. Users of older versions of Helm are still affected by this technique.

New techniques for the threat matrix

1. Initial access

Exposed sensitive interfaces

Exposing a sensitive interface to the internet poses a security risk. Some popular frameworks were not intended to be exposed to the internet, and therefore don’t require authentication by default. Thus, exposing them to the internet allows unauthenticated access to a sensitive interface which might enable running code or deploying containers in the cluster by a malicious actor. Examples of such interfaces that were seen exploited include Apache NiFi, Kubeflow, Argo Workflows, Weave Scope, and the Kubernetes dashboard.

2. Execution

Sidecar injection

A Kubernetes Pod is a group of one or more containers with shared storage and network resources. Sidecar container is a term that is used to describe an additional container that resides alongside the main container. For example, service-mesh proxies are operating as sidecars in the applications’ pods. Attackers can run their code and hide their activity by injecting a sidecar container to a legitimate pod in the cluster instead of running their own separated pod in the cluster.

3. Persistence

Malicious admission controller

Admission controller is a Kubernetes component that intercepts, and possibly modifies, requests to the Kubernetes API server. There are two types of admissions controllers: validating and mutating controllers. As the name implies, a mutating admission controller can modify the intercepted request and change its properties. Kubernetes has a built-in generic admission controller named MutatingAdmissionWebhook. The behavior of this admission controller is determined by an admission webhook that the user deploys in the cluster. Attackers can use such webhooks for gaining persistence in the cluster. For example, attackers can intercept and modify the pod creation operations in the cluster and add their malicious container to every created pod.

4. Credential access

Access managed identity credential

Managed identities are identities that are managed by the cloud provider and can be allocated to cloud resources, such as virtual machines. Those identities are used to authenticate with cloud services. The identity’s secret is fully managed by the cloud provider, which eliminates the need to manage the credentials. Applications can obtain the identity’s token by accessing the Instance Metadata Service (IMDS). Attackers who get access to a Kubernetes pod can leverage their access to the IMDS endpoint to get the managed identity’s token. With a token, the attackers can access cloud resources.

Malicious admission controller

In addition to persistency, a malicious admission controller can be used to access credentials. One of the built-in admission controllers in Kubernetes is ValidatingAdmissionWebhook. Like MutatingAdmissionWebhook, this admission controller is also generic, and its behavior is determined by an admission webhook that is deployed in the cluster. Attackers can use this webhook to intercept the requests to the API server, record secrets, and other sensitive information.

5. Lateral movement

CoreDNS poisoning

CoreDNS is a modular Domain Name System (DNS) server written in Go, hosted by Cloud Native Computing Foundation (CNCF). CoreDNS is the main DNS service that is being used in Kubernetes. The configuration of CoreDNS can be modified by a file named corefile. In Kubernetes, this file is stored in a ConfigMap object, located at the kube-system namespace. If attackers have permissions to modify the ConfigMap, for example by using the container’s service account, they can change the behavior of the cluster’s DNS, poison it, and take the network identity of other services.

ARP poisoning and IP spoofing

Kubernetes has numerous network plugins (Container Network Interfaces or CNIs) that can be used in the cluster. Kubenet is the basic, and in many cases the default, network plugin. In this configuration, a bridge is created on each node (cbr0) to which the various pods are connected using veth pairs. The fact that cross-pod traffic is through a bridge, a level-2 component, means that performing ARP poisoning in the cluster is possible. Therefore, if attackers get access to a pod in the cluster, they can perform ARP poisoning, and spoof the traffic of other pods. By using this technique, attackers can perform several attacks at the network-level which can lead to lateral movements, such as DNS spoofing or stealing cloud identities of other pods (CVE-2021-1677).

6. Collection

In this update, we also add a new tactic to the threat matrix: collection. In Kubernetes, collection consists of techniques that are used by attackers to collect data from the cluster or through using the cluster.

Images from private registry

The images that are running in the cluster can be stored in a private registry. For pulling those images, the container runtime engine (such as Docker or containerd) needs to have valid credentials to those registries. If the registry is hosted by the cloud provider, in services like Azure Container Registry (ACR) or Amazon Elastic Container Registry (ECR), cloud credentials are used to authenticate to the registry. If attackers get access to the cluster, in some cases they can obtain access to the private registry and pull its images. For example, attackers can use the managed identity token as described in the “Access managed identity credential” technique. Similarly, in EKS, attackers can use the AmazonEC2ContainerRegistryReadOnly policy that is bound by default to the node’s IAM role.

Protect your containerized environments

Understanding the attack surface of containerized environments is the first step of building security solutions for these environments. The revised threat matrix for Kubernetes can help organizations identify the current gaps in their defenses’ coverage against the different threats that target Kubernetes.

We recommend that you start protecting your containerized environment with Azure Defender today. Learn more about Azure Defender’s support for container security.

To learn more about Microsoft Security solutions visit our website. Bookmark the Security blog to keep up with our expert coverage on security matters. Also, follow us at @MSFTSecurity for the latest news and updates on cybersecurity.

The post Secure containerized environments with updated threat matrix for Kubernetes appeared first on Microsoft Security Blog.

In this blog, we’ll reveal a new campaign that was observed recently by ASC that targets Kubeflow, a machine learning toolkit for Kubernetes. We observed that this attack effected on tens of Kubernetes clusters.

Kubeflow is an open-source project, started as a project for running TensorFlow jobs on Kubernetes. Kubeflow has grown and become a popular framework for running machine learning tasks in Kubernetes. Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs. This fact makes Kubernetes clusters that are used for ML tasks a perfect target for crypto mining campaigns, which was the aim of this attack.

During April, we observed deployment of a suspect image from a public repository on many different clusters. The image is ddsfdfsaadfs/dfsdf:99. By inspecting the image’s layers, we can see that this image runs an XMRIG miner:

This repository contains several more images, which differ in the mining configuration. We saw some deployments of those images too.

Looking at the various clusters that the above image ran on showed that most of them run Kubeflow. This fact implies that the access vector in this attacker is the machine-learning framework.

The question is how can Kubeflow be used as an access vector for such an attack?

Kubeflow framework consists of many different services. Some of those services include: frameworks for training models, Katib and Jupyter notebook server, and more.

Kubeflow is a containerized service: the various tasks run as containers in the cluster. Therefore, if attackers somehow get access to Kubeflow, they have multiple ways to run their malicious image in the cluster.

The framework is divided into different namespaces, which are a collection of Kubeflow services. Those namespaces are translated into Kubernetes namespaces in which the resources are deployed.

In first access to Kubeflow, the user is prompted to create a namespace:

In the picture above, we created a new namespace with the default name anonymous. This namespace is broadly seen in the attack and was one of the indicators to the access vector in this campaign.

Kubeflow creates multiple CRDs in the cluster which expose some functionality over the API server:

In addition, Kubeflow exposes its UI functionality via a dashboard that is deployed in the cluster:

The dashboard is exposed by Istio ingress gateway, which is by default accessible only internally. Therefore, users should use port-forward to access the dashboard (which tunnels the traffic via the Kubernetes API server).

In some cases, users modify the setting of the Istio Service to Load-Balancer which exposes the Service (istio-ingressgateway in the namespace istio-system) to the Internet. We believe that some users chose to do it for convenience: without this action, accessing to the dashboard requires tunneling through the Kubernetes API server and isn’t direct. By exposing the Service to the Internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.

If attackers have access to the dashboard, they have multiple methods to deploy a backdoor container in the cluster. We will demonstrate two options:

1. Kubeflow enables users to create a Jupyter notebook server. Kubeflow allows users to choose the image for the notebook server, including an option to specify a custom image:

This image doesn’t necessarily have to be a legitimate notebook image, thus attackers can run their own image using this feature.

2. Another method that attackers can use is to deploy a malicious container from a real Jupyter notebook: attackers can use a new or existing notebook for running their Python code. The code runs from the notebook server, which is a container by itself with a mounted service account. This service account (by default configuration) has permissions to deploy containers in its namespace. Therefore, attackers can use it to deploy their backdoor container in the cluster. Here’s an example of deploying a container from the notebook using its service account:

The Kubernetes threat matrix that we recently published contains techniques that can be used by attackers to attack the Kubernetes cluster. A representation of this campaign in the matrix would look like:

The attacker used an exposed dashboard (Kubeflow dashboard in this case) for gaining initial access to the cluster. The execution and persistence in the cluster were performed by a container that was deployed in the cluster. The attacker managed to move laterally and deploy the container using the mounted service account. Finally, the attacker impacted the cluster by running a cryptocurrency miner.

How to check if your cluster is impacted?

1. Verify that the malicious container is not deployed in the cluster. The following command can help you to check it:

kubectl get pods –all-namespaces -o jsonpath=”{.items[*].spec.containers[*].image}”  | grep -i ddsfdfsaadfs 

2. In case Kubeflow is deployed in the cluster, make sure that its dashboard isn’t exposed to the internet: check the type of the Istio ingress service by the following command and make sure that it is not a load balancer with a public IP:

kubectl get service istio-ingressgateway -n istio-system

Conclusion

Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector: an exposed service to the internet. However, this is the first time that we have identified an attack that targets Kubeflow environments specifically.

When deploying a service like Kubeflow within a cluster it is crucial to be aware of security aspects such as:

1. Authentication and access control to the application.
2. Monitor the public-facing endpoints of the cluster. Make sure that sensitive interfaces are not exposed to the internet in an unsecure method. You can restrict public load balancers in the cluster by using Azure Policy, which now has integration with Gatekeeper.
3. Regularly monitor the runtime environment. This includes monitoring the running containers, their images, and the processes that they run.
4. Allow deployments of only trusted images and scan your images for vulnerabilities. The allowed images in the cluster can be restricted by using Azure Policy.

To learn more about AKS Support in Azure Security Center, please see this documentation.

Start a trial of Azure Security Center Standard to get advanced threat protection capabilities.

The post Misconfigured Kubeflow workloads are a security risk appeared first on Microsoft Security Blog.

Updated on May 10, 2021: An updated version of the threat matrix for containers is available here.

Kubernetes, the most popular container orchestration system and one of the fastest-growing projects in the history of open source, becomes a significant part of many companies’ compute stack. The flexibility and scalability of containers encourage many developers to move their workloads to Kubernetes. While Kubernetes has many advantages, it also brings new security challenges that should be considered. Therefore, it is crucial to understand the various security risks that exist in containerized environments, and specifically in Kubernetes.

The MITRE ATT&CK® framework is a knowledge base of known tactics and techniques that are involved in cyberattacks. Started with coverage for Windows and Linux, the matrices of MITRE ATT&CK cover the various stages that are involved in cyberattacks (tactics) and elaborate the known methods in each one of them (techniques). Those matrices help organizations understand the attack surface in their environments and make sure they have adequate detections and mitigations to the various risks. MITRE ATT&CK framework tactics include:

• Initial access
• Execution
• Persistence
• Privilege escalation
• Defense evasion
• Credential access
• Discovery
• Lateral movement
• Impact

When we in Azure Security Center started to map the security landscape of Kubernetes, we noticed that although the attack techniques are different than those that target Linux or Windows, the tactics are actually similar. For example, a translation of the first four tactics from OS to container clusters would look like 1. “initial access to the computer” becomes “initial access to the cluster”, 2. “malicious code on the computer” becomes “malicious activity on the containers”, 3. “maintain access to the computer” becomes “maintain access to the cluster”, and 4. “gain higher privileges on the computer” becomes “gain higher privileges in the cluster”.

Therefore, we have created the first Kubernetes attack matrix: an ATT&CK-like matrix comprising the major techniques that are relevant to container orchestration security, with focus on Kubernetes.

As can be seen, the matrix contains the 9 tactics listed above. Each one of them contains several techniques that can be used by attackers to achieve different goals. Below are the descriptions of each one of the techniques.

1. Initial Access

The initial access tactic consists of techniques that are used for gaining access to the resource. In containerized environments, those techniques enable first access to the cluster. This access can be achieved directly via the cluster management layer or, alternatively, by gaining access to a malicious or vulnerable resource that is deployed on the cluster.

• Using cloud credentials

In cases where the Kubernetes cluster is deployed in a public cloud (e.g., AKS in Azure, GKE in GCP, or EKS in AWS), compromised cloud credential can lead to cluster takeover. Attackers who have access to the cloud account credentials can get access to the cluster’s management layer.

• Compromised images in registry

Running a compromised image in a cluster can compromise the cluster. Attackers who get access to a private registry can plant their own compromised images in the registry. The latter can then be pulled by a user. In addition, users often use untrusted images from public registries (such as Docker Hub) that may be malicious.

Building images based on untrusted base images can also lead to similar results.

• Kubeconfig file

The kubeconfig file, also used by kubectl, contains details about Kubernetes clusters including their location and credentials. If the cluster is hosted as a cloud service (such as AKS or GKE), this file is downloaded to the client via cloud commands (e.g., “az aks get-credential” for AKS or “gcloud container clusters get-credentials” for GKE).

If attackers get access to this file, for instance via a compromised client, they can use it for accessing the clusters.

• Vulnerable application

Running a public-facing vulnerable application in a cluster can enable initial access to the cluster. A container that runs an application that is vulnerable to remote code execution vulnerability (RCE) may be exploited. If service account is mounted to the container (default behavior in Kubernetes), the attacker will be able to send requests to the API server using this service account credentials.

• Exposed dashboard

The Kubernetes dashboard is a web-based user interface that enables monitoring and managing a Kubernetes cluster. By default, the dashboard exposes an internal endpoint (ClusterIP service). If the dashboard is exposed externally, it can allow unauthenticated remote management of the cluster.

2. Execution

The execution tactic consists of techniques that are used by attackers to run their code inside a cluster.

• Exec into container

Attackers who have permissions, can run malicious commands in containers in the cluster using exec command (“kubectl exec”). In this method, attackers can use legitimate images, such as an OS image (e.g., Ubuntu) as a backdoor container, and run their malicious code remotely by using “kubectl exec”.

• New container

Attackers may attempt to run their code in the cluster by deploying a container. Attackers who have permissions to deploy a pod or a controller in the cluster (such as DaemonSet \ ReplicaSet\ Deployment) can create a new resource for running their code.

• Application exploit

An application that is deployed in the cluster and is vulnerable to a remote code execution vulnerability, or a vulnerability that eventually allows code execution, enables attackers to run code in the cluster. If service account is mounted to the container (default behavior in Kubernetes), the attacker will be able to send requests to the API server using this service account credentials.

• SSH server running inside container

SSH server that is running inside a container may be used by attackers. If attackers gain valid credentials to a container, whether by brute force attempts or by other methods (such as phishing), they can use it to get remote access to the container by SSH.

3. Persistence

The persistence tactic consists of techniques that are used by attackers to keep access to the cluster in case their initial foothold is lost.

• Backdoor container

Attackers run their malicious code in a container in the cluster. By using the Kubernetes controllers such as DaemonSets or Deployments, attackers can ensure that a constant number of containers run in one, or all, the nodes in the cluster.

• Writable hostPath mount

hostPath volume mounts a directory or a file from the host to the container. Attackers who have permissions to create a new container in the cluster may create one with a writable hostPath volume and gain persistence on the underlying host. For example, the latter can be achieved by creating a cron job on the host.

• Kubernetes CronJob

Kubernetes Job is a controller that creates one or more pods and ensures that a specified number of them successfully terminate. Kubernetes Job can be used to run containers that perform finite tasks for batch jobs. Kubernetes CronJob is used to schedule Jobs. Attackers may use Kubernetes CronJob for scheduling execution of malicious code that would run as a container in the cluster.

4. Privilege escalation

The privilege escalation tactic consists of techniques that are used by attackers to get higher privileges in the environment than those they currently have. In containerized environments, this can include getting access to the node from a container, gaining higher privileges in the cluster, and even getting access to the cloud resources.

• Privileged container

A privileged container is a container that has all the capabilities of the host machine, which lifts all the limitations regular containers have. Practically, this means that privileged containers can do almost every action that can be performed directly on the host. Attackers who gain access to a privileged container, or have permissions to create a new privileged container (by using the compromised pod’s service account, for example), can get access to the host’s resources.

• Cluster-admin binding

Role-based access control (RBAC) is a key security feature in Kubernetes. RBAC can restrict the allowed actions of the various identities in the cluster. Cluster-admin is a built-in high privileged role in Kubernetes. Attackers who have permissions to create bindings and cluster-bindings in the cluster can create a binding to the cluster-admin ClusterRole or to other high privileges roles.

• hostPath mount

hostPath mount can be used by attackers to get access to the underlying host and thus break from the container to the host. (See “3: Writable hostPath mount” for details).

• Access cloud resources

If the Kubernetes cluster is deployed in the cloud, in some cases attackers can leverage their access to a single container in order to get access to other cloud resources outside the cluster. For example, in AKS each node contains service principal credential that is stored in /etc/kubernetes/azure.json. AKS uses this service principal to create and manage Azure resources that are needed for the cluster operation.

By default, the service principal has contributor permissions in the cluster’s Resource Group. Attackers who get access to this service principal file (by hostPath mount, for example) can use its credentials to access or modify the cloud resources.

5. Defense evasion

The defense evasion tactic consists of techniques that are used by attackers to avoid detection and hide their activity.

• Clear container logs

Attackers may delete the application or OS logs on a compromised container in an attempt to prevent detection of their activity.

• Delete Kubernetes events

A Kubernetes event is a Kubernetes object that logs state changes and failures of the resources in the cluster. Example events are a container creation, an image pull, or a pod scheduling on a node.

Kubernetes events can be very useful for identifying changes that occur in the cluster. Therefore, attackers may want to delete these events (e.g., by using: “kubectl delete events–all”) in an attempt to avoid detection of their activity in the cluster.

• Pod / container name similarity

Pods that are created by controllers such as Deployment or DaemonSet have random suffix in their names. Attackers can use this fact and name their backdoor pods as they were created by the existing controllers. For example, an attacker could create a malicious pod named coredns-{random suffix} which would look related to the CoreDNS Deployment.

Also, attackers can deploy their containers in the kube-system namespace where the administrative containers reside.

• Connect from proxy server

Attackers may use proxy servers to hide their origin IP. Specifically, attackers often use anonymous networks such as TOR for their activity. This can be used for communicating with the applications themselves or with the API server.

6. Credential access

The credential access tactic consists of techniques that are used by attackers to steal credentials.

In containerized environments, this includes credentials of the running application, identities, secrets stored in the cluster, or cloud credentials.

• List Kubernetes secrets

A Kubernetes secret is an object that lets users store and manage sensitive information, such as passwords and connection strings in the cluster. Secrets can be consumed by reference in the pod configuration. Attackers who have permissions to retrieve the secrets from the API server (by using the pod service account, for example) can access sensitive information that might include credentials to various services.

• Mount service principal

When the cluster is deployed in the cloud, in some cases attackers can leverage their access to a container in the cluster to gain cloud credentials. For example, in AKS each node contains service principal credential. (See “4: Access cloud resources” for more details.)

• Access container service account

Service account (SA) represents an application identity in Kubernetes. By default, an SA is mounted to every created pod in the cluster. Using the SA, containers in the pod can send requests to the Kubernetes API server. Attackers who get access to a pod can access the SA token (located in /var/run/secrets/kubernetes.io/serviceaccount/token) and perform actions in the cluster, according to the SA permissions. If RBAC is not enabled, the SA has unlimited permissions in the cluster. If RBAC is enabled, its permissions are determined by the RoleBindings \ ClusterRoleBindings that are associated with it.

• Application credentials in configuration files

Developers store secrets in the Kubernetes configuration files, such as environment variables in the pod configuration. Such behavior is commonly seen in clusters that are monitored by Azure Security Center. Attackers who have access to those configurations, by querying the API server or by accessing those files on the developer’s endpoint, can steal the stored secrets and use them.

7. Discovery

The discovery tactic consists of techniques that are used by attackers to explore the environment to which they gained access. This exploration helps the attackers to perform lateral movement and gain access to additional resources.

• Access the Kubernetes API server

The Kubernetes API server is the gateway to the cluster. Actions in the cluster are performed by sending various requests to the RESTful API. The status of the cluster, which includes all the components that are deployed on it, can be retrieved by the API server. Attackers may send API requests to probe the cluster and get information about containers, secrets, and other resources in the cluster.

• Access Kubelet API

Kubelet is the Kubernetes agent that is installed on each node. Kubelet is responsible for the proper execution of pods that are assigned to the node. Kubelet exposes a read-only API service that does not require authentication (TCP port 10255). Attackers with network access to the host (for example, via running code on a compromised container) can send API requests to the Kubelet API. Specifically querying https://[NODE IP]:10255/pods/ retrieves the running pods on the node. https://[NODE IP]:10255/spec/ retrieves information about the node itself, such as CPU and memory consumption.

• Network mapping

Attackers may try to map the cluster network to get information on the running applications, including scanning for known vulnerabilities. By default, there is no restriction on pods communication in Kubernetes. Therefore, attackers who gain access to a single container, may use it to probe the network.

• Access Kubernetes dashboard

The Kubernetes dashboard is a web-based UI that is used for monitoring and managing the Kubernetes cluster. The dashboard allows users to perform actions in the cluster using its service account (kubernetes-dashboard) with the permissions that are determined by the binding or cluster-binding for this service account. Attackers who gain access to a container in the cluster, can use its network access to the dashboard pod. Consequently, attackers may retrieve information about the various resources in the cluster using the dashboard’s identity.

• Instance Metadata API

Cloud providers provide instance metadata service for retrieving information about the virtual machine, such as network configuration, disks, and SSH public keys. This service is accessible to the VMs via a non-routable IP address that can be accessed from within the VM only. Attackers who gain access to a container, may query the metadata API service for getting information about the underlying node. For example, in Azure, the following request would retrieve all the metadata information of an instance: http:///metadata/instance?api-version=2019-06-01

8. Lateral movement

The lateral movement tactic consists of techniques that are used by attackers to move through the victim’s environment. In containerized environments, this includes gaining access to various resources in the cluster from a given access to one container, gaining access to the underlying node from a container, or gaining access to the cloud environment.

• Access cloud resources

Attackers may move from a compromised container to the cloud environment. (See “4: Access cloud resources” for details).

• Container service account

Attackers who gain access to a container in the cluster may use the mounted service account token for sending requests to the API server, and gaining access to additional resources in the cluster. (See “6: Access container service account” for more details.)

• Cluster internal networking

Kubernetes networking behavior allows traffic between pods in the cluster as a default behavior. Attackers who gain access to a single container may use it for network reachability to another container in the cluster.

• Applications credentials in configuration files

Developers store secrets in the Kubernetes configuration files, for example, as environment variables in the pod configuration. Using those credentials attackers may gain access to additional resources inside and outside the cluster. (See “6: Application credentials in configuration files” for more details.)

• Writable volume mounts on the host

Attackers may attempt to gain access to the underlying host from a compromised container. (See “3: Writable hostPath mount” for more details.)

• Access Kubernetes dashboard

Attackers who have access to the Kubernetes dashboard may manage the cluster resources and also run their code on the various containers in the cluster using the built-in “exec” capability of the dashboard. (See “7: Access Kubernetes dashboard” for more details.)

• Access tiller endpoint

Helm is a popular package manager for Kubernetes maintained by CNCF. Tiller is the server-side component of Helm up to version 2.

Tiller exposes internal gRPC endpoint in the cluster, listens to port 44134. By default, this endpoint does not require authentication. Attackers may run code on any container that is accessible to the tiller’s service and perform actions in the cluster, using the tiller’s service account, which often has high privileges.

9. Impact

The Impact tactic consists of techniques that are used by attackers to destroy, abuse, or disrupt the normal behavior of the environment.

• Data destruction

Attackers may attempt to destroy data and resources in the cluster. This includes deleting deployments, configurations, storage, and compute resources.

• Resource hijacking

Attackers may abuse a compromised resource for running tasks. A common abuse is to use compromised resources for running digital currency mining. Attackers who have access to a container in the cluster or have permissions to create new containers may use them for such activity.

• Denial of service

Attackers may attempt to perform a denial of service attack, which makes the service unavailable to the legitimate users. In container clusters, this include attempts to block the availability of the containers themselves, the underlying nodes, or the API server.

Understanding the attack surface of containerized environments is the first step of building security solutions for these environments. The matrix that was presented above can help organizations identify the current gaps in their defenses’ coverage against the different threats that target Kubernetes. Azure Security Center can help you protect your containers environment. Learn more about Azure Security Center’s support for container security.

The post Threat matrix for Kubernetes appeared first on Microsoft Security Blog.