These incidents demonstrate the importance of securing code before it is deployed. Defenses are shifting left to the earliest stages of the application development. In addition to the development phase, we must make sure DevOps environments are secured as well.<\/p>\n\n\n\n
At Microsoft, we have conducted extensive research into the techniques that malicious adversaries may use to attack DevOps environments. We categorized these techniques into their related tactics and mapped these into a threat matrix. This mapping aims to help defenders to better understand the landscape and possible attacker actions, so defenders are better equipped to defend against each technique and protect DevOps environments.<\/p>\n\n\n\n
DevOps threat matrix<\/h2>\n\n\n\n
Our goal for developing the threat matrix for DevOps is to build a comprehensive knowledgebase that defenders can use to keep track of and build defenses against relevant attack techniques. Using the MITRE ATT&CK<\/a> framework as a base, we collected techniques and attack vectors associated with DevOps environments and created a matrix dedicated to DevOps attack methods.<\/p>\n\n\n\n It is worth noting that the tactics in this matrix must be looked at from the DevOps perspective. For example, execution techniques in a Virtual Machine running Windows or Linux OS are different from the Execution in a DevOps pipeline. In the Linux case, execution means running code in the OS. When we talk about DevOps environments, it means running code in the pipeline or DevOps resources. In addition to using this threat matrix to categorize attacks and corresponding defense methods, defenders can work alongside red teams to continuously test assumptions and find new potential attack techniques.<\/p>\n\n\n\n The initial access tactic refers to techniques an attacker may use for gaining access to the DevOps resources \u2013 repositories, pipelines, and dependencies. The following techniques may be a precondition for the next steps:<\/p>\n\n\n\n SCM authentication \u2013 <\/strong>Access by having an authentication method to the organization\u2019s source code management. It may be a personal access token (PAT), an SSH key, or any other allowed authentication credential. An example of a method an attacker can use to achieve this technique is using a phishing attack against an organization.<\/p>\n\n\n\n CI\/CD service authentication \u2013 <\/strong>Similar to SCM authentication, an attacker can leverage authentication to the CI\/CD service in order to attack the organization\u2019s DevOps.<\/p>\n\n\n\n Organization\u2019s public repositories \u2013 <\/strong>Access to the organization\u2019s public repositories that are configured with CI\/CD capabilities. Depending on the organization\u2019s configuration, these repositories may have the ability to trigger a pipeline run after a pull request (PR) is created.<\/p>\n\n\n\n Endpoint compromise \u2013 <\/strong>Using an existing compromise, an attacker can leverage the compromised developer\u2019s workstation, thus gaining access to the organization\u2019s SCM, registry, or any other resource the developer has access to.<\/p>\n\n\n\n Configured webhooks \u2013 <\/strong>When an organization has a webhook configured, an attacker could use it as an initial access method into the organization\u2019s network by using the SCM itself to trigger requests into that network. This could grant the attacker access to services that are not meant to be publicly exposed, or that are running old and vulnerable software versions inside the private network.<\/p>\n\n\n\n The execution tactic refers to techniques that may be used by a malicious adversary to gain execution access on pipeline resources \u2013 the pipeline itself or the deployment resources. Some of the techniques in this section contain explanations about the different ways to perform them, or what we call sub-techniques:<\/p>\n\n\n\n Poisoned pipeline execution (PPE) \u2013 <\/strong>Refers to a technique where an attacker can inject code to an organization\u2019s repository, resulting in code execution in the repository\u2019s CI\/CD system. There are different sub-techniques to achieve code execution:<\/p>\n\n\n\n Dependency tampering \u2013 <\/strong>Refers to a technique where an attacker can execute code in the DevOps environment or production environment by injecting malicious code into a repository\u2019s dependencies. Thus, when the dependency is downloaded, the malicious code is executed. Some sub-techniques that can be used to achieve code execution include:<\/p>\n\n\n\n DevOps resources compromise \u2013 <\/strong>Pipelines are, at the core, a set of compute resources executing the CI\/CD agents, alongside other software. An attacker can target these resources by exploiting a vulnerability in the OS, the agent\u2019s code, other software installed in the VMs, or other devices in the network to gain access to the pipeline.<\/p>\n\n\n\n Control of common registry \u2013<\/strong> An attacker can gain control of a registry used by the organization, resulting in malicious images or packages executed by the pipeline VMs or production VMs.<\/p>\n\n\n\n The persistency tactic consists of different techniques that an attacker may use for maintaining access to a victim environment:<\/p>\n\n\n\n Changes in repository \u2013 <\/strong>Adversaries can use the automatic tokens from inside the pipeline to access and push code to the repository (assuming the automatic token has enough permissions to do so). They can achieve persistency this way using several sub-techniques:<\/p>\n\n\n\n Inject in Artifacts \u2013<\/strong> some CI environments have the functionality for creating artifacts to be shared between different pipeline executions. For example, in GitHub we can store artifacts and download them using a GitHub action from the pipeline configuration.<\/p>\n\n\n\n Modify images in registry \u2013<\/strong> In cases where the pipelines have permissions to access the image registry (for example, for writing back images to the registry after build is done) the attacker could modify and plant malicious images in the registry, which would continue to be executed by the user\u2019s containers.<\/p>\n\n\n\n Create service credentials \u2013<\/strong> A malicious adversary can leverage the access they already have on the environment and create new credentials for use in case the initial access method is lost. This could be done by creating an access token to the SCM, to the application itself, to the cloud resources, and more.<\/p>\n\n\n\n The privilege escalation techniques are used by an attacker to elevate the privileges in the victim\u2019s environment, gaining higher privileges for already compromised resources:<\/p>\n\n\n\n Secrets in private repositories \u2013 <\/strong>Leveraging an already gained initial access method, an attacker could scan private repositories for hidden secrets. The chances of finding hidden secrets in a private repo are higher than in a public repository, as, from the developer\u2019s point of view, this is inaccessible from outside the organization.<\/p>\n\n\n\n Commit\/push to protected branches \u2013 <\/strong>The pipeline has access to the repository that may be configured with permissive access, which could allow to push code directly to protected branches, allowing an adversary to inject code directly into the important branches without team intervention.<\/p>\n\n\n\n Certificates and identities from metadata services \u2013 <\/strong>Once an attacker is running on cloud-hosted pipelines, the attacker could access the metadata services from inside the pipeline and extract certificates (requires high privileges) and identities from these services.<\/p>\n\n\n\n Credential access techniques are used by an attacker to steal credentials:<\/u><\/strong><\/p>\n\n\n\n User<\/strong> credentials<\/strong> \u2013<\/strong> In cases where the customer requires access to external services from the CI pipeline (for example, an external database), these credentials reside inside the pipeline (can be set by CI secrets, environment variables, etc.) and could be accessible to the adversary.<\/p>\n\n\n\n Service<\/strong> credentials \u2013<\/strong> There are cases where the attacker can find service credentials, such as service-principal-names (SPN), shared-access-signature (SAS) tokens, and more, which could allow access to other services directly from the pipeline.<\/p>\n\n\n\n The lateral movement tactic refers to techniques used by attackers to move through different resources. In CI\/CD environments, this may refer to moving to deployment resources, to build artifacts and registries, or to new targets.<\/p>\n\n\n\n Compromise build artifacts \u2013<\/strong> As in other supply chain attacks, once the attacker has control of the CI pipelines, they can interfere with the build artifacts. This way, malicious code could be injected into the building materials before building is done, hence injecting the malicious functionality into the build artifacts.<\/p>\n\n\n\n Registry injection \u2013 <\/strong>If the pipeline is configured with a registry for the build artifacts, the attacker could infect the registry with malicious images, which later would be downloaded and executed by containers using this registry.<\/p>\n\n\n\n Spread to deployment resources \u2013 <\/strong>If the pipeline is configured with access to deployment resources, then the attacker has the same access to these resources, allowing the attacker to spread. This could result in code execution, data exfiltration and more, depending on the permissions granted to the pipelines.<\/p>\n\n\n\n Defense evasion techniques could be used by attackers to bypass defenses used in a DevOps environment and allow attacks to continue under the radar:<\/p>\n\n\n\n Service logs manipulation \u2013 <\/strong>Service logs enable defenders to detect attacks in their environment. An attacker running inside an environment (for example, in the build pipelines) could change the logs to prevent defenders from observing the attack. This is similar to an attacker changing the history logs on a Linux machine, preventing any observer from seeing the commands executed by the attacker.<\/p>\n\n\n\n Compilation manipulation \u2013 <\/strong>if an attacker wishes to leave no traces in the SCM service, the attacker may change the compilation process in order to inject the malicious code. This may be done in several ways:<\/p>\n\n\n\n Reconfigure branch protections \u2013 <\/strong>Branch protection tools allow an organization to configure steps before a PR\/commit is approved into a branch. Once an attacker has admin permissions, they may change these configurations and introduce code into the branch without any user intervention.<\/p>\n\n\n\n The impact tactic refers to the techniques an attacker could use for exploiting access to the CI\/CD resources for malicious purposes, and not as another step in the attack, as these techniques could be noisy and easy to detect:<\/p>\n\n\n\n DDoS \u2013 <\/strong>An adversary could use the compute resources they gained access to in order to execute distributed denial of services (DDoS) attacks on external targets.<\/p>\n\n\n\n Cryptocurrency mining \u2013 <\/strong>The compute resources could be used for crypto mining controlled by an adversary.<\/p>\n\n\n\n Local DoS \u2013 <\/strong>Once the attacker is running on the CI pipelines, the attacker can perform a denial service attack from said pipelines to customers by shutting down agents, rebooting, or by overloading the VMs.<\/p>\n\n\n\n Resource deletion \u2013 <\/strong>An attacker with access to resources (cloud resources, repositories, etc.) could permanently delete the resources to achieve denial of services.<\/u><\/strong><\/p>\n\n\n\n The exfiltration tactic refers to different techniques that could be used by an attacker to exfiltrate sensitive data from victim environment:<\/p>\n\n\n\n Clone private repositories \u2013 <\/strong>Once attackers have access to CI pipelines, they also gain access to the private repositories (for example, the GITHUB_TOKEN can be used in GitHub), and therefore could clone and access the code, thus gaining access to private IP.<\/p>\n\n\n\n Pipeline logs \u2013 <\/strong>An adversary could access the pipeline execution logs, view the access history, the build steps, etc. These logs may contain sensitive information about the build, the deployment, and in some cases even credentials to services, to user accounts and more.<\/p>\n\n\n\n Exfiltrate data from production resources \u2013 <\/strong>In cases where the pipelines can access the production resources, the attackers will have access to these resources as well. Therefore, they can abuse this access for exfiltrating production data.<\/p>\n\n\n\n The threat matrix we\u2019ve presented here is meant to help defenders gain better understanding of the attack surface associated with DevOps environments. Better understanding of these threats will help Microsoft and other security vendors build stronger defenses and implement better coverage against these threats.<\/p>\n\n\n\n Organizations can use this matrix to map the weak spots in their own infrastructure and harden their environments, including the following steps:<\/p>\n\n\n\n The final step would be building a better organization culture for DevOps security and keeping the environment secure. <\/a><\/p>\n\n\n\n![\"DevOps](\"https:\/\/www.microsoft.com\/en-us\/security\/blog\/wp-content\/uploads\/2023\/04\/Figure-1.-Matrix-1024x582.png\")
1. Initial access<\/h3>\n\n\n\n
2. Execution<\/h3>\n\n\n\n
\n
\n
3. Persistence<\/h3>\n\n\n\n
\n
4. Privilege escalation<\/h3>\n\n\n\n
5. Credential access<\/h3>\n\n\n\n
6. Lateral movement<\/h3>\n\n\n\n
7. Defense evasion<\/h3>\n\n\n\n
\n
8. Impact<\/h3>\n\n\n\n
9. Exfiltration<\/h3>\n\n\n\n
Summary<\/u><\/strong><\/h2>\n\n\n\n
\n