Toward trusted sensing for the cloud: Introducing Project Freta

Editor’s note, Feb. 14, 2024 – The Project Freta analysis web portal is no longer publicly accessible. Please contact project-freta@microsoft.com.

“Sunlight is said to be the best of disinfectants.”

                             ―Louis D. Brandeis, 1914

We often think about the field of computer security as a field of walls and barriers that keep intruders out. With Project Freta, we invite readers to think not of walls but of sunlight. When attackers build malware that cannot be detected, they gain enormous economic value. Undetected malware can be continuously re-used: it is never part of attack reporting, never summons incident responders, and never alerts the victim to a data theft event. The economics of reuse can justify enormous attacker investment in malware non-discoverability. Conversely, once a malware strain is discovered, its value plummets in tandem with its reusability. In this stealth economy, that which hides in darkness is removed with sunlight. The question for defenders, then, is how can we raise the cost of non-discovery? Is there a point beyond which a class of malware is no longer economically viable?

Incubated at Microsoft Research, Project Freta is a roadmap toward trusted sensing for the cloud that can allow enterprises to engage in regular, complete discovery sweeps for undetected malware. The project’s namesake, Warsaw’s Freta Street, was the birthplace of Marie Curie, a pioneer of battlefield imaging. While snapshot-based memory forensics is a field now in its second decade, no commercial cloud has yet provided customers the ability to perform full memory audits of thousands of virtual machines (VMs) without intrusive capture mechanisms and a priori forensic readiness. Just as yesteryear’s film cameras and today’s smartphones have similar megapixels but vastly different ease of use and availability, Project Freta intends to automate and democratize VM forensics to a point where every user and every enterprise can sweep volatile memory for unknown malware with the push of a button—no setup required.

The goal of this democratization effort is to increase the development cost of undiscoverable cloud malware toward its theoretical maximum. What would happen if a commercial cloud could guarantee the capture of malware, no matter how expensive or exotic, in volatile memory? Producers of stealthy malware would then be locked into an expensive cycle of complete re-invention, rendering such a cloud an unsuitable place for cyberattacks. This is the future we wish to realize.

To this end, we propose four properties of trusted sensing to maximize malware discovery and present our technical work along this roadmap to date. As a technology demonstration, Project Freta is opening public access to an analysis portal capable of automatically fingerprinting and auditing a memory snapshot of most cloud-based Linux VMs; over 4,000 kernel versions are supported automatically. Hyper-V checkpoint files captured from a modern enterprise can be searched for everything from cryptominers to advanced kernel rootkits. This prototype previews an exciting future option for cloud consumers: transitioning from boutique forensic consulting services to automated malware discovery built into the bedrock of a commercial cloud.

Unbiased data from armored sensors

In computer security we strive to be evidence-driven. Unfortunately, using reports of cyberattacks to guide defensive approaches is subject to a well-known yet powerful survivor bias that can distort the importance of data and lead us astray.

Abraham Wald’s work during WWII provided a famously repeated example of survivor bias in a dataset: reports of bullet holes in airplanes. Rather than recommend armor where bullet holes were reported, Wald recommended armor only in areas where there were no reports of bullet holes. This was due to a critical insight about the nature of bullet hole reporting: bullet holes were only counted when an airplane arrived home. Successful attacks removed the attack from the dataset, hence successful attacks could be described only by their absence from the dataset. No reports of bullet attacks on the engines? Armor the engines.

Today’s computer security sensors suffer from this same bias. When attackers obtain a model of our sensors and design a way to evade these sensors, we receive no reports of cyberattacks. It is tempting to look at areas in which we receive few or no attack reports and think: “I’ve received no reports of successful attacks on my attack reporting capability!” This statement is not as reassuring as it sounds. While we are flooded with billions of reports of malware a year, it is important to understand that synthesizing or mutating existing malware is a nearly cost-free exercise.

Meanwhile, attackers that manage to strike directly at a security sensor, security agent, or privileged operating system component can disable one step in the attack reporting pipeline and vanish without a trace, leaving only an absence in our attack reporting. The lowest-resourced attackers dominate our datasets; the highly resourced live outside our datasets forever. We must invert this data.

Project Freta was designed and built with survivor bias at its core. It is a security project designed from first principles to drive the cost of sensor evasion as high as possible and in many cases render evasion technically infeasible. To achieve this, in July of 2018 we started with a clean slate.

A Greenfield Mandate

The fact that sensor evasion is possible at all is surprising to many outsiders to the field of computer security. Anyone who’s visited a retail store understands that it’s a good idea to put the security cameras out of reach—why haven’t we done something similar in the cloud? The answer is a familiar one: backwards compatibility. Our first networked computers didn’t have the silicon real estate to devote to an isolated security sensor. Opportunities for technology companies to break backwards compatibility and “greenfield” redesign with security in mind have appeared only periodically, seen primarily in the mobile and console industries. These redesigns have allowed for increasing hardware separation between the compute plane and the security plane, a detailed topic for platform security journals beyond the scope of this blog post. In today’s cloud, this separation between compute and security planes usually occurs at the hypervisor; tenant workloads are separated from provider workloads via the hypervisor barrier. Is the hypervisor barrier strong enough to prevent sensor evasion? Maybe not.

Recently, microarchitectural flaws and forensics research have called some of the properties of the hypervisor barrier into question. A recent forensics research paper, “Who Watches The Watcher? Detecting Hypervisor Introspection from Unprivileged Guests” presented at DFRWS in 2018, articulated a method that attackers resident in the compute plane could employ to detect when they are being observed from the security plane—piercing the hypervisor barrier and allowing for pre-emptive self-destruct in order to avoid discovery. This paper came complete with an open-source implementation of the technique. This meant that even Virtual Machine Introspection (VMI) endpoint sensors could be evaded with existing open-source software.

Implementing and democratizing trusted sensing for the cloud meant first articulating the properties of a system that would be immune to these types of attacks:

To achieve these properties and end the stealthy-malware arms race, incrementally improving existing endpoint technology is insufficient. When attackers and defenders share a microarchitecture, every detection move a defender makes disturbs the environment in a way that is eventually discoverable by an attacker invested in secrecy. The only way to discover such attackers is to remove their insight into defense. This left the question: how much engineering was required to fully automate memory forensics, operate at efficiencies that enabled cloud-scale processing, and still retain the element of surprise?

Building Project Freta

A brief technology evaluation taught the team that starting from scratch was the only viable approach. To leave no place to hide, we needed to accept the huge data footprint imposed by whole-system memory analysis. To address detection and burn, such as “red pill attacks,” we needed two components: 1) an offline analysis system that could operate in batch mode and 2) a sensor that could provide whole-system memory captures without executing a single clarifying instruction on the guest. Finally, to mitigate sabotage we needed to ensure our system was built from the ground up to be memory safe.

For Project Freta, many of these challenges compounded to make “instant forensics for everyone” a daunting task:

• Untouchable Images: Many existing forensic approaches execute clarifying instructions on the guest, such as copying KASLR keys. Unfortunately, these instructions can tip off malware to a capture event. The requirement not to interact with the target OS, needed to ensure the element of surprise, mandated a forensic imaging technology that was completely “blind.” As a consequence, memory scrambled by security mechanisms such as ASLR needed to be decoded without keys or context. This task is complex enough for one operating system, and it’s a templating nightmare to support any operating system. Project Freta now supports over 4,000 Linux kernels.
• Universal OS Support: The long-standing forensics requirement that information about the operating system be arrived at a priori needed to be removed. This meant quickly fingerprinting any operating system in the cloud given only a scrambled memory image. We knew from the beginning that, given private symbols, this could be achieved for Windows in a believable way. So, we chose Linux instead, knowing that the large number of publicly available kernels for Linux would make this problem significantly more difficult. It also meant that a functional result would pay down the technology debt required to build faith in the approach. With Linux behind us, Windows support is on our roadmap.
• Cloud Scale: Automated capture and analysis won’t matter to customers if a day of cloud compute is needed to perform a single audit. To operate on modern cloud enterprises, we knew that the ability to programmatically audit 100,000 machines in a short, cost-bounded timeframe was a minimum requirement. This meant architecting from the beginning for batch processing in the cloud, including OS fingerprinting in the performance requirements, and thinking ahead about edge cases such as high-performance-compute VMs with 100+ gigabytes of RAM.
• Memory Safety: We knew that any system designed to hunt for tools fielded by the most well-resourced attackers would itself become a target. Given the history and preponderance of memory-corruption exploits, we made the choice as a team to embrace Rust at the beginning, architecting the entire capability from scratch in Rust from line one and building upon no existing software. This has yielded a high-performance analysis engine for memory images of arbitrary size that also has memory safety properties built in.

Project Freta: This release

The Project Freta analysis engine consumes snapshots of whole-system Linux volatile memory and extracts an enumeration of system objects. Some kernel hooking identification is performed automatically; this can be used by analysts to detect novel rootkits. The analysis portal is available in prototype form for public use: https://freta.microsoft.com.

The prototype portal supports many types of memory snapshots as inputs. Currently, only a Hyper-V checkpoint has been evaluated to provide a reasonable approximation of the “element of surprise” necessary to achieve trusted sensing:

• Use the Hyper-V checkpoint feature to produce a VMRS file
• Convert a VMWare snapshot to produce a CORE file
• Extract memory from within a running system using AVML
• Extract memory from within a running system using LiME

Once the snapshot is uploaded to the portal and analyzed, the report data is made available via the portal and both REST and Python APIs. Project Freta’s initial release supports API-driven automated use.

The report contains an enumeration of system objects over the interval during which the sample was taken:

• Global values and addresses
• Debugged processes
• In-memory files
• Kernel interrupt table
• Kernel modules
• Kernel syscall table
• Networks
• Open files
• ARP table (arp)
• Open sockets
• Processes
• Unix sockets (lsof)

Of interest to defenders: debugging relationships are provided to allow for investigation of counter-debugging techniques; library imports are listed to allow for investigation of LD_PRELOAD based attacks; and simple hooking of systems calls is detected and mapped.

For further info, please visit our documentation.

Project Freta: Futures

Project Freta’s second component for achieving trusted sensing is a sensor built for Azure that allows operators to migrate the volatile memory of live virtual machines to an offline analysis environment without disrupting execution. Completed in the winter of 2019, this sensor capability is currently only available to Microsoft researchers and is not fielded to any of our commercial clouds—executive briefings and demos are available. This sensor, coupled with the Freta analysis environment, demonstrates a path to cheap, automated memory forensic audits of large enterprises (10,000+ VMs).

A great deal of development lies ahead for Project Freta: adding support for Windows, extending our automated program analysis capabilities, and experimenting with AI-based decision-making for novel threat detection. For now, we are opening access to the analysis portal for customer use and experimentation. We hope that Project Freta empowers administrators and responders and is used globally as it has been used at Microsoft: to hunt advanced intruders and their toolkits. We welcome your feedback at project-freta@microsoft.com.